Deep Neural Networks led to major breakthroughs in artificial intelligence. This unreasonable effectiveness is explained in part by a scaling-up in terms of computing power, available datasets and model size -the latter was achieved by building deeper and deeper networks. In this thesis, recognizing that such models are hard to comprehend and to train, we study the set of neural networks under the prism of their functional equivalence classes in order to group networks by orbits and to only manipulate one carefully selected representant. Based on these theoretical considerations, we propose a variant of the stochastic gradient descent (SGD) algorithm which amounts to inserting, between the SGD iterations, additional steps allowing us to select the representant of the current equivalence class that minimizes a certain energy. The redundancy of the network's parameters highlighted in the first part naturally leads to the question of the efficiency of such networks, hence to the question of their compression. We develop a novel method, iPQ, relying on vector quantization that drastically reduces the size of a network while preserving its accuracy. When combining iPQ with a new pre-conditioning technique called Quant-Noise that injects quantization noise in the network before its compression, we obtain state-of-the-art tradeoffs in terms of size/accuracy. Finally, willing to confront such algorithms to product constraints, we propose an application allowing anyone to make an ultra-low bandwidth video call that is deployed on-device and runs in real time.

Je tiens tout d'abord à remercier les deux rapporteurs, Raquel Urtasun et Francois Malgouyres, pour avoir pris le temps de relire ce manuscrit, pour leurs retours et leurs conseils. Merci également à l'ensemble des membres du jury pour leur lecture en profondeur du manuscrit et leurs questions. C'est un honneur de pouvoir soutenir ma thèse devant un panel de chercheurs aussi accomplis et renommés.

Merci à Hervé de m'avoir donné la chance de m'exprimer à travers le stage puis la thèse. La confiance immédiate et l'autonomie grisante que tu m'as accordés m'ont construit en tant que chercheur et m'ont indéniablement fait grandir. Un grand merci pour ton accompagnement et ton soutien décisif dans tous les moments clés de cette thèse, aussi bien au niveau scientifique, relationnel que personnel.

Merci à toi Rémi d'avoir soutenu mes velléités en toutes circonstances, qu'elles soient mathématiques ou plus appliquées. Merci d'avoir calmé mes doutes lorsqu'il le fallait et de m'avoir canalisé. Je retiendrai ta simplicité et ton accessibilité, ta volonté intransigeante de compréhension adossée à une conception du temps long en recherche et ta capacité à compiler, de tête, plusieurs pages de preuve.

Merci à Benjamin pour tes fulgurances qui ont bootstrapé ma thèse. Merci à Gabriel Peyré qui a été parmi les premiers à m'ouvrir les portes intrigantes et exaltantes de la recherche. Merci Angela d'avoir bien voulu former un duo de compression avec moi, sous l'égide d'Armand, Hervé et Edouard. Merci à Moustapha Cissé pour m'avoir montré qu'il était possible d'écrire un article en un temps record. Merci à Matthijs pour ta disponibilité et ton aide précieuse sur l'implémentation de kernels. Merci à la team FaceGen de m'avoir accueilli avec bienveillance il y a plus d'un an et demi et de m'avoir donné l'opportunité de construire une première démo. Merci à toi Maxime, j'ai énormément appris lors de nos interactions quasi-quotidiennes et je retiens avant tout ta confiance et ton écoute. Merci à Camille, Daniel et Onur : vous vous reconnaitrez sans doute dans quelques illustrations de cette thèse.

Résumé

Les réseaux de neurones profonds sont à l'origine de percées majeures en intelligence artificielle. Ce succès s'explique en partie par un passage à l'échelle en termes de puissance de calcul, d'ensembles de données d'entrainement et de taille des modèles considérés -le dernier point ayant été rendu possible en construisant des réseaux de plus en plus profonds. Dans cette thèse, partant du constat que de tels modèles sont difficiles à appréhender et à entrainer, nous étudions l'ensemble des réseaux de neurones à travers leurs classes d'équivalence fonctionnelles, ce qui permet de les grouper par orbites et de ne manipuler qu'un représentant bien choisi. Ces considérations théoriques nous ont permis de proposer une variante de l'algorithme de descente de gradient stochastique qui consiste à insérer, au cours des itérations, des étapes permettant de choisir le représentant de la classe d'équivalence courante minimisant une certaine énergie. La redondance des paramètres de réseaux profonds de neurones mise en lumière dans ce premier volet amène naturellement à la question de l'efficience de tels réseaux, et donc de leur compression. Nous développons une nouvelle méthode de compression, appelée iPQ et reposant sur de la quantification vectorielle, prouvant qu'il est possible de réduire considérablement la taille d'un réseau tout en préservant sa capacité de prédiction. En combinant iPQ avec une procédure de pré-conditionnement appelée Quant-Noise qui consiste à injecter du bruit de quantification dans le réseau avant sa compression, nous obtenons des résultats état de l'art en termes de compromis taille/capacité de prédiction. Voulant confronter nos recherches à des contraintes de type produit, nous proposons enfin une application de ces algorithmes permettant un appel vidéo à très faible bande passante, déployée sur un téléphone portable et fonctionnant en temps réel.

5

Merci à Facebook et à toute l'équipe de FAIR Paris. Merci au PhD Squad et en particulier à Neil, Pauline, Alexis, Guillaume, Timothée, Alexandre, Alexandre, Léonard, Nicolas, Louis Martin et les autres pour leur soutien par temps de deadline, pour les parties endiablées de baby-foot agrémentées de gamelles et autres snakes, et pour les moments de respiration hors du temps à Yosemite, en offsite ou en conférence. Merci à Pauline Luc de m'avoir accompagné pour ma première conférence au pays des biergarten. Merci à Timothée Lacroix et Alexandre Sablayrolles pour nos escapades enseignantes et culinaires au Rwanda. Merci à Ezéchiel et Clément pour nos discus-sions lyriques sur la science en général. Merci à Marianne d'avoir fait irruption au début de ma thèse, merci pour ta joie de vivre et pour tous nos gouters sur le rooftop qui me manquent déjà. Merci enfin à Yana pour ton amitié indéfectible, pour toutes nos discussions et nos sessions de télétravail agrémentées de crêpes.

Merci à tous mes professeurs de mathématiques et de physique du lycée et de prépa pour m'avoir montré ce que la raison est capable de construire et pour m'avoir transmis la passion des sciences. Mon parcours est indissociable de ces rencontres et cette thèse n'aurait pas existé sans votre enseignement.

Merci à ma famille et en particulier à mes parents, qui m'ont tout donné pour que je m'épanouisse. Merci à Clara et Arthur pour leur soutien et pour leur écoute patiente lorsque je leur résumais un article. Merci à Margaux d'avoir illuminé ma thèse et d'avoir partagé mes joies et mes peines, tu es mon ancrage dans ce monde.

Une pensée particulière pour ma grand-mère maternelle, qui voulait que je sois préfet, et à qui je dédie cette thèse.

Notation

We first define a neural network using the formalism of graph theory to disentangle the architecture from the values of the weights. Unless mention of the contrary, we consider networks with ReLU non-linearities defined as 𝜎(𝑥) = max(𝑥, 0) for 𝑥 ∈ R.

Architecture

A neural network architecture can be represented as a particular directed acyclic graph 𝐺 = (𝑉, 𝐸). We denote a neuron by 𝜈 ∈ 𝑉 and a connection by 𝑒 = 𝜈 → 𝜇 ∈ 𝐸. Each neuron 𝜈 belongs to a layer ℓ(𝜈) ∈ 0, 𝐿 .

• If ℓ(𝜈) = 0 then 𝜈 belongs to the input layer.

• If 0 < ℓ(𝜈) < 𝐿 then 𝜈 belongs to one of the 𝐿 -1 hidden layers.

• If ℓ(𝜈) = 𝐿 then 𝜈 belongs to the output layer.

𝐺 is such that two connected neurons necessarily belong to consecutive layers. We denote 𝐻 ⊂ 𝑉 the set of all hidden neurons. Note that two neurons 𝜇 and 𝜈 in consecutive layers may be connected or not. We denote the neurons of layer ℓ by

𝑁 ℓ {𝜈 | ℓ(𝜈) = ℓ}.
A full path 𝑝 is a sequence of connected neurons 𝑝 = (𝜈 0 , . . . , 𝜈 𝐿 ) where 𝜈 ℓ ∈ 𝑁 ℓ , 0 ≤ ℓ ≤ 𝐿 and 𝜈 ℓ-1 → 𝜈 ℓ ∈ 𝐸 for 1 ≤ ℓ ≤ 𝐿. We say that a connection 𝑒 belongs to 𝑝 = (𝜈 0 , . . . , 𝜈 𝐿 ) if there exists ℓ such that 𝑒 = 𝜈 ℓ-1 → 𝜈 ℓ . We may write 𝑝 ∩ 𝐻 to denote the hidden neurons (𝜈 1 , . . . , 𝜈 𝐿-1 ) belonging to the path 𝑝. We denote by 𝒫(𝐺) the set of all full paths connecting some input neuron to some output neuron. We also define a partial path 𝑞 = (𝜈 ℓ , . . . , 𝜈 𝐿 ) as a sequence of connected neurons between any hidden layer ℓ with 0 ≤ ℓ ≤ 𝐿 and the output layer. We finally denote by path segment a sequence of connected neurons (𝜈 ℓ , . . . , 𝜈 ℓ ′ ) where 0 ≤ ℓ ≤ 𝐿 and 0 ≤ ℓ ′ ≤ 𝐿. We denote 𝒬(𝐺) the set of such partial paths and we have 𝒫(𝐺) ⊂ 𝒬(𝐺). We may omit the dependency of the underlying graph 𝐺 when it is obvious and simply write 𝒫 and 𝒬. We may simply denote full paths by paths when the context is clear.

For any neuron 𝜈, we define

prev(𝜈) {𝜇 ∈ 𝑉 | 𝜇 → 𝜈 ∈ 𝐸} next(𝜈) {𝜇 ∈ 𝑉 | 𝜈 → 𝜇 ∈ 𝐸} (1) (2)
and for a set of neurons 𝑉 , prev(𝑉 ) = ∪ 𝜈∈𝑉 prev(𝜈). We denote by Parents(𝜈) the set of all parent neurons of 𝜈 Parents(𝜈)

⋃ ℓ prev ℓ ({𝜈}) (3)
where prev ℓ denotes the composition of the operator prev with itself. We similarly define Children(𝜈) the set of all children neurons of 𝜈. We also introduce the notation

• → 𝜈 to denote any edge 𝑒 ∈ 𝐸 that has the form 𝜇 → 𝜈 for some 𝜇 ∈ prev(𝜈) and similarly for the notation 𝜈 → •.

Weights and Biases

Throughout the manuscript, we will manipulate quantities involving the weights and biases of the network, and find it cleaner to index them all using the connections of the network or its neurons. The graph 𝐺 = (𝑉, 𝐸) is valued with the weights of the network. The weights can be represented:

• At the connection level by 𝑤 𝑒 ∈ R with 𝑒 ∈ 𝐸;

• At the layer level by 𝑊 (ℓ) ∈ R 𝑁 ℓ-1 ×𝑁 ℓ where 𝑊 (ℓ) = (𝑤 𝜈→𝜇 ) 𝜈,𝜇∈𝑁 ℓ-1 ×𝑁 ℓ and ℓ ∈ 1, 𝐿 ;

• At the network level by 𝑤 ∈ R 𝐸 .

In order to define weights at the layer level, we write by convention 𝑤 𝜈→𝜇 = 0 if 𝜈 → 𝜇 / ∈ 𝐸. Besides, given a fixed architecture 𝐺, we allow the case where some weights 𝑤 𝑒 , 𝑒 ∈ 𝐸 are zero. Similarly to the weights, the biases can be represented:

• At the neuron level by 𝑏 𝜈 ∈ R for any hidden neuron 𝜈 ∈ 𝐻 ∪ 𝑁 𝐿 ;

• At the layer level by 𝑏 (ℓ) ∈ R 𝑁 ℓ where 𝑏 (ℓ) = (𝑏 𝜈 ) 𝜈∈𝑁 ℓ and ℓ ∈ 1, 𝐿 ;

• At the network level by 𝑏 ∈ R 𝐻∪𝑁 𝐿 .

We denote the global network parameterization by 𝜃 = (𝑤, 𝑏) and refer to elements of 𝜃 as parameters of the network. Networks with at least one hidden layer are such that 𝐿 ≥ 2. The case 𝐿 = 1 corresponds to a linear layer without any non-linearity. Note that a network has 𝐿 affine layers and 𝐿 -1 hidden layers. Finally, we define useful support and sign sets as follows:

supp(𝑤) {𝑒 ∈ 𝐸 | 𝑤 𝑒 ̸ = 0} ⊂ 𝐸 supp(𝑏) {𝜈 ∈ 𝐻 ∩ 𝑁 𝐿 | 𝑏 𝜈 ̸ = 0} ⊂ 𝐻 ∩ 𝑁 𝐿 supp(𝜃) supp(𝑤) ∪ supp(𝑏) ⊂ 𝐸 ∪ (𝐻 ∩ 𝑁 𝐿 )
We further define the extended sign operator as follows. For 𝑥 ∈ R, sign(𝑥) = 1 if 𝑥 > 0, sign(𝑥) = 0 if 𝑥 = 0 and sign(𝑥) = -1 if 𝑥 < 0. When applied to a vector or a matrix, sign is taken pointwise. We finally define

Sign 𝑤 {𝑤 ′ ∈ R 𝐸 | sign(𝑤 ′ ) = sign(𝑤)} Supp 𝑤 {𝑤 ′ ∈ R 𝐸 | supp(𝑤 ′ ) ⊆ supp(𝑤)}
Similarly, we define Sign 𝑏 and Supp 𝑏 and denote Sign 𝜃 Sign 𝑤 × Sign 𝑏 and we denote Supp 𝜃 Supp 𝑤 × Supp 𝑏

Function

We will also need to manipulate the output function or intermediary functions implemented by the network. Let 𝐺 be a fixed architecture 𝐺 valued with 𝜃. Recall that we denote by 𝜎 the ReLU non-linearity. We define:

• Layer-wise functions. A neural network can be recursively implemented using intermediary row vector functions 𝑦 (ℓ) (𝜃) : R 𝑁 0 → R 𝑁 ℓ for ℓ ∈ 0, 𝐿 . We define 𝑦 (0) (𝜃, 𝑥) = 𝑥 and, for ℓ ∈ 1, 𝐿 -1 , 𝑦 (ℓ) (𝜃, 𝑥) = 𝜎 ( 𝑦 (ℓ-1) (𝑥)𝑊 (ℓ) + 𝑏 (ℓ) )

.

(4)

• Output functions. The function implemented by the whole network is 𝑦 (𝐿) (𝜃, 𝑥) = 𝑦 (𝐿-1) (𝜃, 𝑥)𝑊 (𝐿) + 𝑏 (𝐿) (5)

i.e. the last layer is an affine function of the previous one. We use the notation 𝑅 𝐺|𝜃 = 𝑦 (𝐿) (𝜃) and we call 𝑅 𝐺|𝜃 the realization of the network architecture 𝐺 given the parameters 𝜃. We write 𝑅 𝐺|𝜃 (𝑥) to denote the evaluation of the defined function at any input 𝑥 ∈ R 𝑁 0 . When the dependency on the graph 𝐺 is obvious, we may simply write 𝑅 𝜃 and 𝑅 𝜃 (𝑥). With a slight abuse of notation and for the sake of clarity, we may also write 𝑅(𝜃, 𝑥).

• Neuron functions. Given a neuron 𝜈 belonging to layer ℓ, we denote the function implemented by 𝜈 before the non-linearity as 𝑦 𝜈 (𝜃) : R 𝑁 0 → R such that 𝑦 𝜈 (𝜃) = 𝑦 (ℓ) 𝜈 (𝜃). ( 6)

Given a fixed architecture 𝐺 valued with respective weights 𝜃 or 𝜃 ′ , we say that 𝜃 and 𝜃 ′ are functionally equivalent if the realizations 𝑅 𝜃 and 𝑅 𝜃 ′ are the same, i.e., if for all 𝑥 ∈ 𝑅 𝑁 0 , 𝑅 𝜃 (𝑥) = 𝑅 𝜃 ′ (𝑥).

Useful Quantities

We denote the value of a full path by 𝑣 𝑝 (𝜃) = 𝑤 𝜈 0 →𝜈 1 . . . 𝑤 𝜈 𝐿-1 →𝜈 𝐿 and we define the activation status of a full path 𝑝 given the parameters 𝜃 and the input 𝑥 as 𝑎 𝑝 (𝜃, 𝑥)

∏ 𝜈∈𝑝∩𝐻 1(𝑦 𝜈 (𝜃, 𝑥) > 0). ( 7
)
We naturally extend the notion of partial path value and the notion of partial path activation status. As the value of a full or partial path only depends on the weights 𝑤 and not on the biases, we may write indifferently 𝑣 𝑝 (𝜃) or 𝑣 𝑝 (𝑤) for any full or partial path. For any path segments 𝑞 = (𝜈 ℓ , . . . , 𝜈 ℓ ′ ), 𝑞 ′ = (𝜇 ℓ ′ , . . . , 𝜇 ℓ ′′ ) such that 𝜈 ℓ ′ = 𝜇 ℓ ′ , we denote the concatenation of 𝑞 and 𝑞 ′ as 𝑞 + 𝑞 ′ = (𝜈 ℓ , . . . , 𝜈 ℓ ′ , 𝜇 ℓ ′ +1 , . . . , 𝜇 ℓ ′′ ).

Algebraic Tools

We will rely on algebraic and geometric interpretation to understand the action of rescaling operations on a parameterization 𝜃. To this end, we represent the mapping between edges and paths by the linear operator P : R 𝐸 → R 𝒫 such that for every connection 𝑒 ∈ 𝐸 and every path 𝑝 ∈ 𝒫,

(P𝛿 𝑒 ) 𝑝 ⎧ ⎨ ⎩ 1 if 𝑒 ∈ 𝑝 0 otherwise (8)
where 𝛿 𝑒 ∈ R 𝐸 is the dirac vector for edge 𝑒. We denote by 𝒟(𝑁 ) the set of diagonal matrices 𝐷 ∈ R 𝑁 ×𝑁 such that, for all 𝑖, 𝑑 𝑖 = 𝐷 𝑖,𝑖 is strictly positive.

Chapter 1 Introduction

Computer Science has shaped our modern society in a tremendous way, in part since the seminal work of Alan Turing, who invented an abstract computer called the Turing Machine in 1936. Since then, this concept has materialized in the form of processors and chips with an extremely wide range of applications. In a fast-paced search for performance, the building blocks of modern computers called transistors have become smaller and smaller according to Moore's law, which states that the number of transistors on a chip doubles after a short, constant period of time1 . Thanks to this miniaturization, the ubiquity of interconnected portable computers, prophesied by Silicon Valley entrepreneurs [START_REF] Gates | Road Ahead[END_REF] or even by french writer Marguerite Duras 2 in 1985 is now a reality.

Such powerful and portable devices, including smartphones or virtual/augmented reality headsets, constitute a fertile ground for a particular class of algorithms called Deep Neural Networks (DNNs). These models are programmed to learn to perform specific tasks -hence the name Deep Learning -and belong to the more general field of Artificial Intelligence, also pioneered by Turing 3 . While DNNs are increasingly powerful for detecting persons in images or understanding and translating speech or text for instance, they still lack efficiency in terms of size and speed.

Hence, after the miniaturization of the computers themselves, the miniaturization or compression of DNNs is now a key challenge to deploy them on-device and in real time. It would offload the servers that currently run such models, reduce the latency and promote more privacy since personal data would be analyzed directly on-device. This is particularly relevant after the European Union's new data privacy and security law entered into force in 2016, the General Data Protection Regulation (GDPR) 4 .

In this Chapter, we first detail the general topics of efficiency and redundancy in deep learning and the underlying reasons motivating our work in Section 1.1. We then briefly summarize the interesting challenges that the Deep Learning community is facing on these areas in Section 1.2 and finally detail our contributions, both academic and product-oriented and the general organization of the manuscript in Section 1.3.

Motivation

We summarize here the main reasons that conducted us to study the redundancy and efficiency of Neural Networks both theoretically and in practice.

The Deep Learning Revolution. Following pioneering work by [START_REF] Rumelhart | Learning internal representations by error propagation[END_REF], [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF] or (LeCun et al., 1998b), the inception of AlexNet by [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] marked a turning point in the development of Deep Learning. Back in 2012, a heavy neural network, trained with stochastic gradient descent (SGD) and properly regularized, surpassed all existing methods by a large margin on the ImageNet competition [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. Since then, the concept of "Ima-geNet moment" was transposed to various domains where Deep Learning percolated with an unreasonable effectiveness, from Natural Language Processing [START_REF] Ruder | NLP's ImageNet moment has arrived[END_REF] a few years ago to protein folding as we write these lines [START_REF] Jumper | High accuracy protein structure prediction using deep learning[END_REF]. Strong empirical evidence of deep learning approaches was also shown for extremely various tasks such as symbolic mathematics [START_REF] Lample | Deep learning for symbolic mathematics[END_REF], fast MRI reconstruction [START_REF] Zbontar | fastMRI: An open dataset and benchmarks for accelerated MRI[END_REF] or quantum physics (Schütt et al., 2017).

Bigger, Hence Better Models. Fueled by this tremendous success on numerous tasks, a fast-paced search for performance is occurring in the research community. Since one straightforward way to improve expressivity -hence performance -is to act on depth by stacking more layers [START_REF] Telgarsky | Benefits of depth in neural networks[END_REF][START_REF] Raghu | On the expressive power of deep neural networks[END_REF], researchers are considering the biggest possible networks they can train [START_REF] Simonyan | Very deep convolutional networks for largescale image recognition[END_REF]He et al., 2015b), now up to 175 billion parameters [START_REF] Rajbhandari | Zero: Memory optimizations toward training trillion parameter models[END_REF][START_REF] Rasley | Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters[END_REF][START_REF] Tom | Language models are few-shot learners[END_REF]. Training such networks requires huge amounts of data (labelled or not), energy and performant distributed computing infrastructure relying on GPGPUs or TPUs 5 .

Redundancy and Efficiency in Deep Learning Models. Then, as the research community produces bigger networks, the question of their redundancy and efficiency naturally arises in order to manipulate such networks more easily. Roughly speaking, redundancy 6 refers to a certain structure in the parameters of a neural network where some weights or groups of weights carry similar information [START_REF] Lecun | Optimal brain damage[END_REF][START_REF] Denil | Predicting parameters in deep learning[END_REF]. On the other hand, efficiency 7 refers to Pareto efficiency [START_REF] Fudenberg | Game Theory[END_REF] of a model given set of metrics 8 , traditionally in terms of model size, model accuracy and inference time (Wang et al., 2018a). Thus, studying redundancy in deep learning models may help improve their efficiency, at least on the model size axis. Note that various setups -especially in terms of hardware -and constraints on these metrics may lead to distinct Pareto optima.

Making the Best Models Available to Everyone. Studying redundancy in deep learning to produce more efficient models is thus crucial for deploying the best models both on servers and on mobile devices. On the server side, the objective is to produce more parsimonious models in terms of parameters [START_REF] Radosavovic | Designing network design spaces[END_REF][START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] or training data [START_REF] Touvron | Training data-efficient image transformers and distillation through attention[END_REF], which may lead to a faster training time 9 , energy savings, or help consider even bigger models to train. On the mobile side, deploying models on embedded devices such as smartphones, autonomous vehicles or virtual/augmented reality headsets 10 opens up to numerous applications. Having such models on-device instead of performing the inference on a remote server reduces the latency and the network congestion, works offline and is compatible with privacy-preserving machine learning where the data stays on the user's device [START_REF] Knott | Crypten: Secure multi-party computation meets machine learning[END_REF], at the cost of exposing the embedded model to various attacks. It also enables federated learning [START_REF] Jakub Konečný | Federated learning: Strategies for improving communication efficiency[END_REF] where a centralized model is trained while training data remains distributed over a large number of client devices with unreliable network connections. 5 Respectively General Purpose Graphics Processing Units and Tensor Processing Units. 6 More details in Sections 2.1 and 2.2. 7 More details in Section 2.3. 8 Such metrics are not independent: larger models are generally more accurate but slower. 9 For instance, AlphaGo [START_REF] Silver | Mastering the game of go without human knowledge[END_REF] took 40 days to train on a vast infrastructure. 10 Such as Oculus Quest 2 for VR (Virtual Reality) or HoloLens 2 for AR (Augmented Reality).

Challenges

Redundancy is intimately related to the concept of over-parameterization, which is a key characteristic of modern neural networks [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF] -see Subsection 2.1.1 for definitions and discussions. Over-parameterized networks have a high ability to fit training data, yet they are challenging to train, they are difficult to comprehend from a theoretical point of view and they pose significant hurdles to real-time applications on embedded devices. We briefly summarize these challenges in this Section and refer the reader to Chapter 2 for a more extensive discussion.

Theoretical Considerations on Redundancy. Parameter redundancy of deep learning models is a well-known fact [START_REF] Denil | Predicting parameters in deep learning[END_REF] and has clear drawbacksfirst and foremost, the model size. However, reducing this redundancy is challenging and indirectly provides hints on the benefits of redundant, over-parameterized models.

At training time, researchers study the interplay between over-parameterization and SGD [START_REF] Li | Learning overparameterized neural networks via stochastic gradient descent on structured data[END_REF][START_REF] Karthik | The impact of neural network overparameterization on gradient confusion and stochastic gradient descent[END_REF] to help mitigate overfitting. Taking advantage of this redundancy may lead to more efficient or performant training procedures. For instance, grouping networks that behave similarly and implicitly performing SGD in a reduced or quotiented space may help, see Chapters 3 and 4.

Redundancy in Practice for Efficient DNNs. While training happens once, the trained model is subsequently used numerous times for inference. For instance, at Facebook, deep learning models analyze trillions of bits of content per day. Then, the challenge is to compress the network -or more generally, to make it more efficient -without losing too much predictive performance or accuracy, sometimes referred to as the size/accuracy tradeoff, see Chapters 5 and 6. Another challenge is to select the best compression algorithm -or combination thereof -among a large set of methods that are not entirely orthogonal, given a target size/accuracy tradeoff. For instance, is it better to compress a large, high-performing network instead of a mobile-efficient architecture that is smaller but has a slightly degraded predictive performance?

On-device and Real Time Deployment. While less redundant networks are generally faster at inference, redundancy is only a part of the story. Indeed, compression algorithms with an excellent compression ratio could necessitate to decompress the network before inference instead of performing the prediction in the compressed domain. Hence, a good compression algorithm also depends on the task and hardware constraints -generally, inference has to be performed in real-time without draining the battery and overloading the RAM 11 , see Chapter 7. This involves low-level considerations on the type of hardware used for inference, see Subsection 2.3.6 for details.

Contributions

We detail the general organization of the manuscript and then enumerate our academic contributions as well as the applications and technology transfers derived directly from our published papers.

Outline

We present here our contributions in ascending order of applicability, measured as the closeness to production. We first start by presenting theoretical contributions on functional equivalence classes, then explore the compression of deep learning models down to the on-device deployment of such efficient models.

Equivalence Classes of Neural Networks. In Chapter 3, we study functional equivalence classes of ReLU neural networks. We first show that such classes contain orbits generated by the action of rescalings and permutations of hidden neurons for networks with arbitrary depth. We then characterize functional equivalence classes for one-hidden-layer networks under some non-degeneracy conditions and investigate the case with many hidden layers by designing algebraic tools to study the problem locally. Leveraging these theoretical considerations, we develop an alternative to the Stochastic Gradient Descent (SGD) algorithm in Chapter 4. Our variant, called Equi-Normalization or ENorm, alternates between standard SGD steps and balancing steps amounting to change the representant of the current functional equivalence class by selecting the one that minimizes a given energy function. Balancing steps preserve the output -hence the accuracy -of the network by definition but modify the gradients of the next SGD step, hence the learning trajectory. In other words, ENorm takes advantage of the redundancy in the parameter space by operating the optimization in the quotient space induced by the functional equivalence relation.

Neural Network Compression.

Studying the redundancy of the network's parameters in the first part of the thesis naturally leads to the question of the efficiency 11 Random Access Memory or RAM is generally used to rapidly store and retrieve working data.

of such networks, hence to the question of their compression. In Chapter 5, we develop a novel method, called Iterative Product Quantization or iPQ, that relies on vector quantization in order to drastically reduce the size of a network while almost preserving its accuracy. The proposed approach iteratively quantizes the layers and then finetunes them. The quantization step is performed by splitting the layer's weight matrix into a set of vectors that are clustered into a common codebook. In order to boost the obtained size/accuracy tradeoffs, we develop a pre-conditioning method that injects carefully selected quantization noise when training the network before its compression. The method, called Quant-Noise, is described in Chapter 6 and has proven to be effective for both iPQ and for traditional scalar quantization such as int8 or int4. Quant-Noise is effective for a variety of tasks and quantization methods and thus reconciles pre-training for both scalar and vector quantization.

Ultra-Low Bandwidth Generative Video Chat. While the compressed size of the network is a significant indicator of the quality of the quantization, other metrics such as inference time and battery usage are also relevant, especially for ondevice, real time applications. Hence, we investigated potential applications for iPQ and Quant-Noise. Among them, we design a method, called FaceGen, to perform ultra-low bandwidth video chats in Chapter 7. FaceGen streams compressed facial landmarks from the sender's phone to the receiver, and uses a generative adversarial network to reconstruct the sender's face based on the stream of landmarks plus one identity embedding sent once at the beginning of the call. The stream of landmarks is compressed to less than 10 kbits/s, and the networks are quantized to a total size of less than 2 MB and run at 20+ frames per second on an iPhone 8.

Publications

The work presented in this manuscript was also published in the following papers, that were written during the thesis.

• Pierre Stock, Benjamin Graham, Rémi Gribonval and Hervé Jégou. Equinormalization of Neural Networks. Published at ICLR 2019 (Stock et al., 2019a). Source code: https://github.com/facebookresearch/enorm.

• Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham and Hervé Jégou. And the Bit Goes Down: Revisiting the Quantization of Neural Networks. Published at ICLR 2020 (Stock et al., 2019b). Source code: https: //github.com/facebookresearch/kill-the-bits.

• Pierre Stock * , Angela Fan * , Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou and Armand Joulin. Training with Quantization Noise for Extreme Model Compression. Published at ICLR 2021 [START_REF] Stock | Training with quantization noise for extreme model compression[END_REF] [START_REF] Oquab | Low bandwidth video-chat compression using deep generative models[END_REF].

The following paper was written during an internship at Facebook AI Research and will not be discussed in this manuscript.

• Pierre Stock, Moustapha Cisse. ConvNets and ImageNet Beyond Accuracy:

Explanations, Bias Detection, Adversarial Examples and Model Criticism. Published at ECCV 2018 [START_REF] Stock | Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases[END_REF].

Technology Transfers

The research presented in this manuscript led to various technology transfers, which are briefly summarized here. Seeking for internal applications and collaborations and delivering product impact was the main focus of the last year of my PhD.

• The iPQ technique described in Chapter 5 is currently used to design fast CPU kernels for quantized models for both server side and mobile side, relying on the fbgemm12 library for quantized matrix multiplication.

• The pre-conditioning technique Quant-Noise (Chapter 6) was used to deploy a quantized int8 model for internal dogfooding. The model aims at detecting harmful content in conversations on-device on real time.

• The low-bandwidth generative video chat method described in Chapter 7 is currently under productionization and led to a US patent application P201451US00.

Chapter 2

Related Work

In this Chapter, we review the lines of work addressing the question of parameter redundancy and efficiency in Deep Learning. In Section 2.1, we first discuss the benefits of depth for neural networks in terms of expressivity and capacity to fit training data. We enumerate the main practical training challenges along with the methods and tools to mitigate them, in particular in terms of normalization layers. This line of work is related to our contribution in Chapter 4, where we re-normalize the network's weights after each training step while preserving the function implemented by the network. Then, we review the theoretical studies aiming at characterizing functional equivalence classes of neural networks in Section 2.2. Such equivalence classes allow to aggregate networks that behave identically to only manipulate one representant per class, thus effectively operating in the quotient space. This work is related to our contribution detailed in Chapter 3 and shows that under some assumptions, the equivalence classes only encompass permutations and rescalings of neurons (see 2.2.1).

Finally, we study network compression in Section 2.3. We review the methods aiming at reducing the redundancy in the set of the network's parameters while maintaining a competitive accuracy and inference speed, in particular scalar and vector quantization. This section is related to our contributions on Iterative Product Quantization and Quantization Noise that are detailed in Chapters 5 and 6. More specifically, the subsection about on-device deployment is related to our Ultra-low Bandwidth Generative Video Chat contribution detailed in Chapter 7.

Over-Parameterization: a Double-Edged Sword

The inception of AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] demonstrated that a deep neural network surpassed existing computer vision techniques by a good margin 1 on the Im-ageNet classification task [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. This success is conditioned on proper training techniques, the availability of large datasets, as well as huge computing capabilities. Since then, a part of the research community has focused on scaling the architectures, the datasets and the training techniques in the search of better performance [START_REF] Tom | Language models are few-shot learners[END_REF]. This fast-paced practical search for deeper networks, followed by a more theoretical analysis of the benefits of depth, is reviewed in Subsection 2.1.1. Then, we briefly enumerate the training challenges posed by such networks in Subsection 2.1.2 along with the tools to mitigate them in Subsection 2.1.3, including various normalization layers. Subsections 2.1.1 and 2.1.2 do not aim to be exhaustive but act rather as motivating illustrations for the remainder of this chapter.

With Greater Depth Comes Greater Expressivity

We briefly review and discuss the notion of over-parameterization, followed by a more theoretical analysis of the benefits of depth in terms of expressivity. Here, we do not aim at a comprehensive review but rather focus on a few illustrative examples.

Over-parameterization

Over-parameterized networks are primarily characterized by their large number of parameters 2 with respect to the number of training samples in the Deep Learning literature [START_REF] Sagun | Empirical analysis of the hessian of over-parametrized neural networks[END_REF][START_REF] Allen-Zhu | A convergence theory for deep learning via over-parameterization[END_REF][START_REF] Li | Learning overparameterized neural networks via stochastic gradient descent on structured data[END_REF]. For instance, AlexNet has 60 million parameters, which is an order of magnitude larger than 1.2 million train images of ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. Note that a more rigorous definition would take into account various factors such as the sample size 3 , the architecture 𝐺 or even the data itself [START_REF] Sagun | Empirical analysis of the hessian of over-parametrized neural networks[END_REF].

Next, we briefly focus on a few illustrative examples acknowledging the parameter redundancy in neural networks in practice. First, the fact that some parameters can 1 The ILSVRC-2012 test top-5 error rate was 15.3% for AlexNet compared to 26.2% for the second best entry of the competition.

2 The number of parameters of a neural network with architecture 𝐺 is the number of connections in 𝐺 (except for the biases). Sometimes authors consider the number of neurons rather than the number of connections [START_REF] Gribonval | Approximation spaces of deep neural networks[END_REF].

3 Fitting 𝑁 training samples in a low dimension input space would require less parameters than fitting 𝑁 training images of standard size 3 × 224 × 224 for instance.

be deleted or pruned without harming the accuracy is well-known [START_REF] Lecun | Optimal brain damage[END_REF], as discussed in Subsection 2.3.1. More recently, [START_REF] Denil | Predicting parameters in deep learning[END_REF] demonstrate that there is a significant redundancy in the network parameters of several deep learning models by accurately predicting 95% of the parameters based only on the remaining 5%, with a minor drop in accuracy. The motivation behind this technique is the fact that the first layer features of a convolutional neural network trained in natural images (e.g. ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF])) tend to be locally smooth with local edge features, similar to local Gabor filters4 . Given this structure, representing the value of each pixel separately is redundant as the value of one pixel is highly correlated with its neighbors. [START_REF] Denil | Predicting parameters in deep learning[END_REF] propose to take advantage of this type of structure to factor the weight matrix. Similar approaches that learn a basis of low-rank filters are explored by [START_REF] Jaderberg | Speeding up convolutional neural networks with low rank expansions[END_REF].

Depth and Expressivity

The relation between the network's depth and its expressivity is widely studied (Pascanu et al., 2013;[START_REF] Montúfar | On the number of linear regions of deep neural networks[END_REF][START_REF] Eldan | The power of depth for feedforward neural networks[END_REF][START_REF] Telgarsky | Benefits of depth in neural networks[END_REF][START_REF] Raghu | On the expressive power of deep neural networks[END_REF][START_REF] Gribonval | Approximation spaces of deep neural networks[END_REF]. For instance, [START_REF] Montúfar | On the number of linear regions of deep neural networks[END_REF] study the number of linear regions defined by a given architecture. As defined more formally in Subsection 2.2.2 in the case of ReLU networks, linear regions are the areas of the input space on which the gradient of the function implemented by the network ∇ 𝑥 𝑅 𝜃 is constant. The number of linear regions is connected to the complexity of the function implemented by the network: more linear regions means that the network is able to fit more complex training data. The authors derive a lower bound on the maximal number of linear regions and show in particular that, for architectures with fixed widths, the maximal number of linear regions -hence the expressivity of the network -grows exponentially in the number of layers.

While the expressivity of a network somehow grows exponentially with its depth, another approach to obtain a network with the same number of parameters is to increase its width when keeping the number of layers fixed. This strategy generally results in less expressive networks, as shown by [START_REF] Eldan | The power of depth for feedforward neural networks[END_REF]. The authors exhibit a simple radial function 𝜙 in R 𝑑 that is implementable by a two-hidden-layer network but cannot be tightly approximated by any one-hidden-layer network, unless its width grows exponentially in the input dimension 𝑑. The authors conclude that "depth can be exponentially more valuable than width" for feedforward networks. 2019). Here the complexity of the considered model (a ResNet-18) is measured as its width parameter, where higher width denotes a network with a larger number of parameters.

Deeper Networks Present Harder Training Challenges

In this Subsection, we briefly review the main training challenges of neural networks that arise as the networks go deeper in the search for more and more expressivity, as mentioned in Subsection 2.1.1. We refer the reader to the cited work below for a more exhaustive overview.

Overfitting

Overfitting is traditionally approached in Machine Learning through the Bias-Variance trade-off, as explained for instance by [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction[END_REF]. According to this principle, as the model complexity -measured with its number of parameters or with more elaborate tools such as the VC dimension [START_REF] Vapnik | Statistical Learning Theory[END_REF] or the Rademacher complexity [START_REF] Bartlett | Rademacher and gaussian complexities: Risk bounds and structural results[END_REF] -increases, the training error decreases while the test error follows a U-shaped curve. Models with low complexity underfit and suffer from high bias whereas models with high complexity exhibit a high variance, suggesting that an intermediate model complexity reaches the optimal trade-off. However, recent work by [START_REF] Belkin | Reconciling modern machine learning practice and the bias-variance trade-off[END_REF], followed by [START_REF] Nakkiran | Deep double descent: Where bigger models and more data hurt[END_REF]; [START_REF] Mei | The generalization error of random features regression: Precise asymptotics and double descent curve[END_REF][START_REF] Stéphane D'ascoli | Double trouble in double descent : Bias and variance(s) in the lazy regime[END_REF] uncovered a double-descent behavior for deep learning models. After a critical regime, the test error goes down again as illustrated in Figure 2-1. This finding is consistent across architectures, optimizers and tasks [START_REF] Nakkiran | Deep double descent: Where bigger models and more data hurt[END_REF], suggesting, as found in practice, that deeper models and more data lead to better performance [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

As stated before, given a fixed training dataset, overfitting is traditionally measured using the number of parameters of the considered model. In the search for more elaborate model complexity measures, [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF] propose a protocol Figure 2-2: Fitting random labels and random pixels on CIFAR10 as illustrated by [START_REF] Zhang | Understanding deep learning requires rethinking generalization[END_REF].

for understanding the effective capacity of machine learning models. The authors train several architectures on a copy of the training data where either (1) the original labels were replaced by random labels; or (2) the pixels were shuffled; or (3) the pixels were drawn randomly from gaussian noise. They observe that the training error goes down to zero provided that the number of epochs is large enough, as depicted in Figure 2-2, and conclude that "deep neural networks easily fit random labels". Or course, if the amount of randomization is larger, the time taken to overfit is longer. Such work paves the way for more formal complexity measures to explain the generalization ability of neural networks.

Vanishing and Exploding Gradients

In the 90's, feedforward neural networks -convolutional or fully connected -were a few layers deep [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF] and were trained through back-propagation. However, [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF] reported difficulties to train recurrent networks in order to learn long-term dependencies -for instance in a sequence of words. The principle of recurrent networks is to apply the same weight matrix iteratively on the input sequence while updating a hidden state. Recurrent networks are usually trained with back-propagation through time (BPTT, [START_REF] Williams | Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity[END_REF]; [START_REF] Rumelhart | Learning internal representations by error propagation[END_REF][START_REF] Paul | Generalization of backpropagation with application to a recurrent gas market model[END_REF], where the network is unfolded for a fixed sequence length and trained using standard back-propagation. When working on an improved version of recurrent networks named LSTMs (Long Short-Term Memory networks, [START_REF] Hochreiter | Long short-term memory[END_REF]), [START_REF] Hochreiter | Gradient flow in recurrent nets: the difficulty of learning long-term dependencies[END_REF] identify two undesirable behaviors of the gradients flowing backward in time: such gradients either blow up or vanish, resulting in training instabilities.

As feedforward networks went deeper, similar vanishing or exploding gradient problems arise [START_REF] Mishkin | All you need is a good init[END_REF] He et al. (2015c). Such training instabilities were related in part to the magnitude of the weights. The research community proposed tools to mitigate this undesirable training behavior as explained in Subsection 2.1.3. Failure modes preventing the training from properly starting were also theoretically studied by [START_REF] Hanin | Which neural net architectures give rise to exploding and vanishing gradients?[END_REF].

The Deep Learning Training Toolbox

The training challenges of deep neural networks mentioned in Subsection 2.1.2 are overcome by various methods designed along the years. We briefly enumerate the main strategies that constitute the toolbox of every researcher and practitioner, in particular in terms of normalization layers. This line of work is related to our contribution detailed in Chapter 4.

Initialization Schemes

Properly initializing the weights of a neural network before training it alleviates the vanishing or exploding gradient problem in the first training iterations and allows stochastic gradient descent algorithms to find a suitable minimum, starting from this initialization. By studying the distribution of activations and gradients, [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] designed an initialization scheme to preserve the variance of activations and gradients across layers for networks with symmetrical activation functions like the sigmoid or the hyperbolic tangent. Following this idea, [START_REF] Mishkin | All you need is a good init[END_REF] and He et al. (2015c) designed initialization schemes for networks with Rectified Linear Units (ReLUs). This leads to a popular weight initialization technique

𝑊 ℓ ∼ 𝒩 ( 0, √ 2/𝑁 ℓ-1 )
where 𝑁 ℓ-1 is the number of input features5 . Finally, failure modes that prevent the training from starting have been theoretically studied by [START_REF] Hanin | How to start training: The effect of initialization and architecture[END_REF].

Data Augmentation

Data augmentation is widely used to easily generate additional data to improve machine learning systems in various areas [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Huang | Deep networks with stochastic depth[END_REF][START_REF] Wu | Detectron2[END_REF] and to reduce overfitting. Traditionally, for object classification, at training time, a random resized crop6 is applied to the image which is then flipped horizontally with a probability 0.5. Since then, many more augmentation techniques were designed. For instance, Zhang et al. (2017a) train a network on convex combinations of pairs of examples and their labels, while [START_REF] Ekin | Autoaugment: Learning augmentation policies from data[END_REF] automatically search for improved data augmentation policies with a method called AutoAugment. We refer the reader to [START_REF] Ekin | Randaugment: Practical automated data augmentation with a reduced search space[END_REF] for a survey of data augmentation techniques. On a side note, the random resized crops used at training time involve a rescaling of the input image, in contrast to the center crops used at test time. Thus, the network is generally presented with larger objects at training time than at test time. This train-test resolution discrepancy is addressed by [START_REF] Touvron | Fixing the traintest resolution discrepancy[END_REF] with a method called FixResNets.

Architectures

As networks are getting deeper, two major architectural changes are introduced to prevent the gradients to vanish. Rectified Linear Units (ReLUs) defined as 𝜎(𝑥) = max(0, 𝑥) are applied pointwise on the activations. [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] were among the first to successfully employ such non-saturating activation functions to Convolutional Neural Networks, as opposed to traditional saturating functions like the sigmoid. Moreover, to allow the information to flow better up and down the network, He et al. (2015a) introduced skip-connections between blocks. More formally, if 𝑓 is a building block7 , adding a skip connection amounts to output 𝑓 (𝑥) + 𝑥 after the block instead of 𝑓 (𝑥) for any input activation 𝑥. Skip connections are now ubiquitous in deep learning architectures such as the Transformers in NLP [START_REF] Vaswani | Attention is all you need[END_REF].

Optimization

Neural networks are originally trained with Stochastic Gradient Descent (SGD)8 , generally with momentum [START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF]. Denoting 𝜃 𝑡 the parameters at time step 𝑡 and ℒ the loss function, one possible set of equations for SGD writes9 :

⎧ ⎨ ⎩ 𝜃 𝑡+1 = 𝜃 𝑡 -𝜂𝑣 𝑡+1 𝑣 𝑡+1 = 𝜇𝑣 𝑡 + ∇ 𝜃 ℒ
where 𝜂 is the learning rate and 𝜇 the momentum coefficient, generally set to 0.9. This remains the main training recipe for Image Classification problems [START_REF] Goyal | Accurate, large minibatch sgd: Training imagenet in 1 hour[END_REF][START_REF] Wu | Group normalization[END_REF]. The learning rate is generally following a schedule, meaning that 𝜂 = 𝜂 𝑡 depends on the current epoch or iteration. A classical schedule starts with a warm-up phase followed by a decay phase 10 (He et al., 2015a). Interestingly, the cosine schedule is gaining traction both in Vision and in NLP [START_REF] Radosavovic | Designing network design spaces[END_REF]. However, choosing a proper learning rate along with its schedule is expensive. Therefore, [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] designed a first-order gradient method named Adagrad that accounts for the anisotropic relation between the network's parameters and the loss function. More precisely, given a default learning rate 𝜂 0 usually set to 0.01 and a small constant for numerical stability 𝜀,

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ 𝑤 𝑒,𝑡+1 = 𝑤 𝑒,𝑡 - 𝜂 0 √ 𝑔 𝑒,𝑡+1 + 𝜀 ∇ 𝑤𝑒 ℒ 𝑔 2 𝑒,𝑡+1 = 𝑔 2 𝑒,𝑡 + (∇ 𝑤𝑒 ℒ) 2 .
In other words, each weight 𝑤 𝑒 is updated with an adaptive learning rate that depends on the sum of the past gradients with respect to this weight. Other adaptive algorithms were derived, or instance by Kingma and Ba (2014) which proposed a variant called Adam, frequently used in NLP. Intuitively, whereas SGD with momentum can be seen as a ball running down a surface, Adam behaves like a heavy ball with friction. For information about second-order methods or convergency considerations, we refer the reader to the manuscript by [START_REF] Bottou | Optimization methods for largescale machine learning[END_REF].

Weight Decay

Weight decay is a regularization method that is widely used in Deep Learning [START_REF] Krogh | A simple weight decay can improve generalization[END_REF]. It adds a penalty term to the traditional loss, for instance the cross-entropy loss 𝒞 in image classification. The training loss writes

ℒ = 𝒞 + 𝜆‖𝜃‖ 2 2
where 𝜃 is the network's parameters (weights and biases) and 𝜆 an hyper-parameter to cross-validate11 . Weight decay is well suited for SGD as it amounts to add a term 2𝜆𝜃 to the gradients computed by back-propagation.

Figure 2-3: The effect of various normalization layers as illlustrated by [START_REF] Wu | Group normalization[END_REF]. 𝑁 denotes the batch size, 𝐶 the channel dimension and 𝐻 and 𝑊 the spatial axes. The colored pixels are normalized with the same mean and standard deviation.

Dropout

Dropout (Hinton et al., 2012b) is a technique that randomly drops neurons 12 , weights or more important structures [START_REF] Ghiasi | Dropblock: A regularization method for convolutional networks[END_REF] during training time with a fixed small probability 𝑝 13 . This prevents the network from overfitting given the large variety of internal states it has to operate on. Dropout serves other purposes, for example, it helps pruning entire layers at test time [START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF] and is also related to our Quant-Noise contribution in Chapter 6.

Normalization Layers

Normalization procedures take an important part in the development of neural networks. While the inputs are almost always preprocessed 14 (LeCun et al., 1989), researchers began to normalize inner features as the networks were deeper. After successful attempts on whitening or local response normalization [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], Batch Normalization (BN) [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] is now a standard normalization layer. Let us detail BN in the context of computer vision. Denote the activations after some layer ℓ in the network by 𝑥 ∈ R 𝑁 ×𝐶×𝐻×𝑊 , where 𝑁 is the batch size, 𝐶 the number of channels, 𝐻 and 𝑊 the respective height and width of the activations, sometimes called spatial dimensions. For instance, an input image generally has shape 1 × 3 × 224 × 224 for three color channels (RGB) and a size of 224 × 224. For simplicity, we often flatten the last two dimensions so that 𝑥 ∈ R 𝑁 ×𝐶×𝐻𝑊 as depicted in Figure 23.

12 When dropping a neuron 𝜈 ∈ 𝐻 during a training iteration, we set 𝑤 •→𝜈 = 0 and 𝑤 𝜈→• = 0 and we do not update 𝑤 •→𝜈 = 0 and 𝑤 𝜈→• during the backward pass. In other words, we detach all the incoming and outcoming connections of 𝜈 from the forward and backward passes.

13 Generally 𝑝 ∈ [0.

√

𝜎 2 𝑐 + 𝜀 where 𝜀 is a small constant for numerical stability and where 𝜇 𝑐 and 𝜎 𝑐 are the sample mean and (biased) standard deviation defined as

𝜇 𝑐 = 1 𝑁 𝐻𝑊 ∑ 𝑛,ℎ,𝑤 𝑥[𝑛, 𝑐, ℎ, 𝑤] 𝜎 2 𝑐 = 1 𝑁 𝐻𝑊 ∑ 𝑛,ℎ,𝑤 (𝑥[𝑛, 𝑐, ℎ, 𝑤] -𝜇 𝑐 ) 2
Second, the BN layer learns an affine transform of ̂ 𝑥 on the channel dimension:

𝑦 = 𝛾 ̂ 𝑥 + 𝛽
where 𝛾 and 𝛽 are learnt parameters of size 𝐶. More formally, for any batch, channel and spatial indexes 𝑛, 𝑐, ℎ, 𝑤, we have

𝑦 𝑛,𝑐,ℎ,𝑤 = 𝛾 𝑐 ̂ 𝑥 𝑛,𝑐,ℎ,𝑤 + 𝛽 𝑐 . (2.1)
Since the normalization statistics 𝜇 𝑐 and 𝜎 𝑐 depend on the batch, at test time BN is switched to evaluation mode and uses fixed statistics 𝜇 𝑐 and 𝜎 𝑐 that are estimated with an exponential moving average of 𝜇 𝑐 and 𝜎 𝑐 during training time. Thus, at test time, BN is an affine layer.

While extremely effective in standard setups, Batch Normalization suffers from known shortcomings. In particular, BN only works well with sufficiently large batch sizes [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF][START_REF] Wu | Group normalization[END_REF]. For batch sizes below 16 or 32, the batch statistics 𝜇 𝑐 and 𝜎 𝑐 have a high variance and the test error increases significantly. Since then, variants of this technique such as Layer, Instance or Group Normalization [START_REF] Lei Ba | Layer normalization[END_REF][START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF][START_REF] Wu | Group normalization[END_REF] were successfully introduced to circumvent the batch dependency, see Figure 2-3 for an illustration. For instance, Transformers [START_REF] Vaswani | Attention is all you need[END_REF] rely on Layer Normalization whereas Generative Adversarial Networks (GANs) use other variants such as the SPADE block [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF]. This line of work is related to our Equi-normalization contribution in Chapter 4 where we re-normalize the weights -not the activations -in order to minimize the global 𝐿 2 norm of the network to ease the training.

Hyperparameter Tuning

One of the main difficulties of neural network training lies in finding the proper set of hyperparameter values 15 in the high-dimensional space of all the possible aforementioned techniques. For instance, [START_REF] Lample | Fader networks: Manipulating images by sliding attributes[END_REF] found it extremely beneficial to add a dropout rate of 0.3 in some part of their architecture and [START_REF] Carion | End-to-end object detection with transformers[END_REF] underline the crucial importance of having two different learning rates for the two main components of their architecture. While the traditional cartesian grid search remains the main investigation tool, it requires a lot of computing power. For instance, with some PhD colleagues, we estimated that the energy consumed on average to produce one deep learning article has the same order of magnitude as the energy required to heat an average household during one year. Some more efficient techniques were developed, such as the gradient-free optimization platform Nevergrad [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF].

Equivalence Classes of Neural Networks

As demonstrated in Section 2.1, appropriate training techniques allow to train deeper and deeper networks in a fast-paced search for performance. Such networks are constructed by stacking elementary layers or more complex building blocks, which amounts to iterative function composition. For instance, the deepest ResNets (He et al., 2015a) have more than 100 layers. Although single-hidden-layer networks are well understood in terms of capacity to approximate functions presenting certain regularity properties 16 [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF][START_REF] Hornik | Approximation capabilities of multilayer feedforward networks[END_REF], deeper networks remain difficult to comprehend despite numerous fructuous attempts [START_REF] Eldan | The power of depth for feedforward neural networks[END_REF][START_REF] Cohen | Convolutional rectifier networks as generalized tensor decompositions[END_REF]. For instance, [START_REF] Mhaskar | Deep vs. shallow networks : An approximation theory perspective[END_REF] prove matching direct and converse approximation theorems of complexity measurement for Gaussian Networks but not for ReLU networks 17 . In this section, we review theoretical studies 15 This traditionally includes the learning rate and learning rate schedule, the optimizer, the momentum, the weight decay, the batch size or the dropout rate to name a few.

16 A known result, proved independently by [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]; [START_REF] Hornik | Approximation capabilities of multilayer feedforward networks[END_REF] states that networks with a single hidden layer with the sigmoid non-linearity can approximate with arbitrary precision any compactly supported continuous function. This result is known as the "Universal Approximation Property" and was extended to ReLU non-linearities for instance [START_REF] Leshno | Multilayer feedforward networks with a nonpolynomial activation function can approximate any function[END_REF].

17 A Gaussian network has 𝑥 ↦ → exp(-𝑥 2 ) as activation function. The claim states as follows.

(1) For a function 𝑓 with a given smoothness, there exists a gaussian network 𝑔 that approximates 𝑓 , the quality of the approximation being controlled by a complexity measure of 𝑔. (2) Reciprocally, if any function 𝑓 is approximated by a gaussian network 𝑔 of given complexity, then the speed at which the approximation error decreases with respect to the complexity of 𝑔 provides information about the smoothness of 𝑓 .

studying deep networks under the light of their functional equivalence classes [START_REF] Héctor | Uniqueness of the weights for minimal feedforward nets with a given input-output map[END_REF][START_REF] Fefferman | Reconstructing a neural net from its output[END_REF]. For ReLU networks, functional equivalence classes are usually described in the parameter domain using so-called permutation and rescaling operations, that we review in Subsection 2.2.1. To better account for the duality between the function implemented by the network (which is abstract to manipulate) and its parameters (which allow for more concrete formulations), we study the Linear Regions (Pascanu et al., 2013) defined in the input space in Subsection 2.2.2. This helps to specify irreducibility or non-degeneracy conditions under which the functional equivalence classes are easily described by the sole rescaling and permutation operations. We review the main known results along with their limitations in Subsection 2.2.3. This work is related to our contribution detailed in Chapter 3. We end this section by discussing some adjacent work in Subsection 2.2.4 and by showcasing some practical applications in Subsection 2.2.5.

Permutations and Rescalings

After introducing some notations, we state the main definitions of rescaling-equivalent parameters in the literature and reconcile them under the condition that the considered parameterizations are admissible -recall that for an admissible parameterization, every hidden neuron is connected to at least one input and one output neuron through a path of non-zero weights (see Notations). We formally show that for ReLU non-linearities, rescalings preserve the function implemented by the network. We then introduce permutation equivalent parameters, that preserve the function implemented by the network for any activation function -and in particular for ReLU networks. In this Section, we fix the network architecture 𝐺. Recall that the function implemented by the parameterization 𝜃 is denoted by 𝑅 𝜃 .

Rescaling Equivalent Parameters

We state three definitions for rescaling-equivalent parameters that we found in the literature for ReLU networks, namely at the layer, neuron or path level and provide an illustration in Figure 2-4. We next show that these definitions are equivalent under the condition that the considered parameterizations are admissible. Such a restriction to admissible parameterization is natural as explained in Subsection 2.2.2 for the case of "dead neurons". The fact that rescaling equivalent parameters preserve the function implemented by the network heavily relies on the homogeneity of the (2.2)

The layer-wise equivalence is present for instance in the work of [START_REF] Nagel | Data-free quantization through weight equalization and bias correction[END_REF], [START_REF] Meller | Same, same but different -recovering neural network quantization error through weight factorization[END_REF], [START_REF] Yuan | Scaling-based weight normalization for deep neural networks[END_REF] or [START_REF] Rolnick | Reverse-engineering deep relu networks[END_REF].

Definition 2.2.1. We denote by 𝒟(𝑁 ) the set of diagonal matrices 𝐷 ∈ R 𝑁 ×𝑁 such that, for all 𝑖, the diagonal element 𝑑 𝑖 = 𝐷 𝑖,𝑖 is strictly positive.

Recall that 𝜃 = (𝑤, 𝑏) can be also viewed as 𝜃 = (𝑊 (1) , . . . , 𝑊 (𝐿) , 𝑏 (1) , 𝑏 𝐿 ).

Definition 2.2.2 (Layer rescaling equivalence). Two parameterizations 𝜃 and 𝜃

′ are rescaling equivalent if, for all ℓ ∈ 1, 𝐿 -1 , there exists 𝐷 (ℓ) ∈ 𝒟(|𝑁 ℓ |) such that, for all ℓ ∈ 1, 𝐿 , 𝑊 ′ (ℓ) = ( 𝐷 (ℓ-1) ) -1 𝑊 (ℓ) 𝐷 (ℓ) and 𝑏 ′ (ℓ) = 𝑏 (ℓ) 𝐷 (ℓ) (2.3)
with the conventions 𝐷 (0) = 𝐼 𝑁 0 and 𝐷 (𝐿) = 𝐼 𝑁 𝐿 .

The neuron-wise equivalence is for instance used by [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF]. Let 𝜈 ∈ 𝐻 be a neuron in some hidden layer and let 𝜆 𝜈 > 0. A neuron-wise scaling is defined as 𝑠 𝜈,𝜆𝜈 : 𝜃 = (𝑤, 𝑏) ↦ → 𝜃 ′ = (𝑤 ′ , 𝑏 ′ ) where for every connection 𝑒 ∈ 𝐸,

𝑤 ′ 𝑒 = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ 𝜆 𝜈 𝑤 𝑒 if 𝑒 = • → 𝜈 1 𝜆𝜈 𝑤 𝑒 if 𝑒 = 𝜈 → • 𝑤 𝑒 otherwise.
(2.4) and where, for every 𝜈 ∈ 𝐻, 𝑏 ′ 𝜈 = 𝜆 𝜈 𝑏 𝜈 . Let 𝒮 be the set of neuron-wise scalings. We observe that two neuron-wise rescalings commute and that every neuron-wise rescaling is invertible, the inverse of 𝑠 𝜈,𝜆𝜈 being 𝑠 𝜈,1/𝜆𝜈 . Let ⟨𝒮⟩ be the commutative group generated by 𝒮. Every 𝑠 ∈ ⟨𝒮⟩ can be uniquely represented as the composition

𝑠 = ○ 𝜈∈𝐻 𝑠 𝜈,𝜆𝜈 (2.5)
where the 𝜆 𝜈 are strictly positive. Note that in this representation, every hidden neuron 𝜈 is associated to exactly one neuron-wise rescaling 𝜆 𝜈 .

Definition 2.2.3 (Neuron rescaling equivalence). Two parameterizations 𝜃 and 𝜃 ′ are rescaling equivalent if there exists 𝑠 ∈ ⟨𝒮⟩ such that 𝜃 ′ = 𝑠(𝜃).

Definition involving paths in the networks are for instance employed by [START_REF] Meng | 𝒢-SGD: Optimizing relu neural networks in its positively scale-invariant space[END_REF] or [START_REF] Yi | Positively scaleinvariant flatness of ReLU neural networks[END_REF]. We define the extended sign operator as, for 𝑥 ∈ R

sign(𝑥) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ 1 if 𝑥 > 0 0 if 𝑥 = 0 -1 otherwise.
(2.6) When applied to a vector or a matrix, sign is taken entry-wise.

Definition 2.2.4 (Path rescaling equivalence). Two parameterizations 𝜃 and 𝜃 ′ are rescaling equivalent if:

(i) sign(𝜃) = sign(𝜃 ′ ) (ii) For every full path 𝑝 ∈ 𝒫, 𝑣 𝑝 (𝜃) = 𝑣 𝑝 (𝜃 ′ ) (iii) For every partial path 𝑞 ∈ 𝒬, 𝑏 𝑞 0 𝑣 𝑞 (𝜃) = 𝑏 ′ 𝑞 0 𝑣 𝑞 (𝜃 ′ ).
Recall that a path 𝑝 = (𝜈 0 , . . . , 𝜈 𝐿 ) ∈ 𝒫 is a sequence of connected neurons from the input to the output layer, whereas a partial path 𝑞 ∈ 𝒬 connects any hidden neuron to the output layer. The value of a path 𝑝 is 𝑣 𝑝 (𝜃) = 𝑤 𝜈 0 →𝜈 1 . . . 𝑤 𝜈 𝐿-1 →𝜈 𝐿 and similarly for any partial path 𝑞 ∈ 𝒬 (see Notations). Our first contribution is to reconcile these definitions on the layer, neuron and path level for admissible parameterizations as proven in Proposition 3.1.1. From now on, we denote by ∼ 𝑆 the rescaling equivalence relation between admissible parameterizations.

Permutation Equivalent Parameters

We provide a definition of permutation equivalent parameterizations (see Figure 2345).

Definition 2.2.5. We denote by Π(𝑁 ) the set of permutations 𝜋 : 1, 𝑁 ↦ → 1, 𝑁 . With a slight abuse of notation, 𝜋 ∈ Π(𝑁 ) also denotes the permutation matrix 𝜋 ∈ R 𝑁 ×𝑁 such that 𝜋 𝑖,𝑗 = 1 if 𝑗 = 𝜋(𝑖) and 0 otherwise. Definition 2.2.6. Two parameterizations 𝜃 = (𝑤, 𝑏) and

𝜃 ′ = (𝑤 ′ , 𝑏 ′ ) are permutation equivalent if, for all ℓ ∈ 1, 𝐿-1 , there exists 𝜋 ∈ Π(|𝑁 ℓ |) such that, for all ℓ ∈ 1, 𝐿 , 𝑊 ′ (ℓ) = ( 𝜋 (ℓ-1) ) -1 𝑊 (ℓ) 𝜋 (ℓ) and 𝑏 ′ (ℓ) = 𝑏 (ℓ) 𝜋 (ℓ) (2.7)
with the conventions 𝜋 (0) = 𝐼 𝑁 0 and 𝜋 (𝐿) = 𝐼 𝑁 𝐿 . We write 𝜃 ∼ 𝑃 𝜃 ′ to denote the permutation equivalence between two parameterizations 𝜃 and 𝜃 ′ .

As illustrated intuitively in Figure 2-5, permutations preserve the function implemented by the network for any non-linearity function, and in particular for ReLU non-linearities. We prove this result more formally in Proposition A.1.2.

Rescaling-permutation Equivalent Parameters

We define permutation-rescaling equivalent and rescaling-equivalent parameters. Definition 2.2.7. We call two parameterizations 𝜃 and 𝜃 ′ permutation-rescaling equivalent and we write 𝜃 ∼ 𝑃 𝑆 𝜃 ′ if there exists a parameterization 𝜃 ′′ such that 𝜃 ∼ 𝑃 𝜃 ′′ and 𝜃 ′′ ∼ 𝑆 𝜃 ′ . We similarly define rescaling-permutation equivalent parameterizations and we denote 𝜃 ∼ 𝑆𝑃 𝜃 ′ if there exists a parameterization 𝜃 ′′ such that

𝜃 ∼ 𝑆 𝜃 ′′ and 𝜃 ′′ ∼ 𝑃 𝜃 ′ .
Next, we show that the rescaling and permutation operations commute in Proposition A.1.3, thus both definitions coincide. From now on, we will denote ∼ 𝑃 𝑆 the permutation-rescaling equivalence. Such transformations preserve the realization of any ReLU network (Proposition 3.1.2).

Sign Flips

We will compare to existing work considering non-ReLU activation functions18 in Subsection 2.2.3. Such activation functions include for instance the hyperbolic tangent, which is not homogeneous. Hence, we define sign flips by adapting Definition 2.2.3. Let 𝒮 ± be the set of neuron-wise scalings restricted for every hidden neuron 𝜈 ∈ 𝑁 to 𝜆 𝜈 ∈ {-1, 1}. Let ⟨𝒮 ± ⟩ be the commutative group generated by 𝒮 ± . Every 𝑠 ∈ ⟨𝒮 ± ⟩ can be uniquely represented as the composition

𝑠 = ○ 𝜈∈𝐻 𝑠 𝜈,𝜆𝜈 (2.8)
where 𝜆 𝜈 = ±1. This operation amounts to flip signs of the weights and associated biases on the parameters.

Definition 2.2.8. Two parameterizations 𝜃 and 𝜃 ′ are sign-flip equivalent if there exists 𝑠 ∈ ⟨𝒮 ± ⟩ such that 𝜃 ′ = 𝑠(𝜃).

Since this Definition will be used only to compare to existing work in Subsection 2.2.3, we do not explore its variants as we did with the rescalings19 . Since tanh is an odd function, sign flips preserve the function implemented by tanh networks but not by ReLU networks. Weights are indicated on the connections between neurons: for instance, 𝑤 𝜇 1 →𝜈 1 = 0. Biases are specified next to the neurons: for instance 𝑏 𝜅 = 0. Input and output neurons are depicted as squares whereas hidden neurons are displayed as circles. Note that the input space is two-dimensional: 𝑥 = (𝑥 𝜇 1 , 𝑥 𝜇 2 ) and the output is scalar: |𝑁 𝐿 | = 1. Right: 3D visualization of the network's realization 𝑥 ↦ → 𝑅 𝜃 (𝑥) which is piecewise affine.

Activation and Linear Regions

In this Subsection, we focus on ReLU networks. Such networks implement piecewise affine realizations 𝑥 ↦ → 𝑅 𝜃 (𝑥) (Pascanu et al., 2013), as depicted for one particular example in Figure 23456. Thus, studying the regions where the gradient is constant -the Linear Regions -yields valuable information about the network's architecture and weights, as uncovered by (Pascanu et al., 2013;[START_REF] Montúfar | On the number of linear regions of deep neural networks[END_REF][START_REF] Raghu | On the expressive power of deep neural networks[END_REF]. The main difficulty lies both in the dimensionality of the input space and in the number of layers 𝐿.

Activation vs. Linear Regions

We first define activation and linear regions of ReLU networks and leverage the work of [START_REF] Hanin | Deep relu networks have surprisingly few activation patterns[END_REF] that precisely explain the distinction between both. As explained in the Notations, for any hidden layer ℓ ∈ 1, 𝐿 -1 , the intermediary function implemented by the layer after the non-linearity 𝜎 writes

𝑦 (ℓ) (𝜃, 𝑥) = 𝜎 ( 𝑦 (ℓ-1) (𝑥)𝑊 (ℓ) + 𝑏 (ℓ) )
.

(2.9)

Recall that the function implemented by a single neuron 𝜈 ∈ 𝐻 is denoted as 𝑦 𝜈 (𝜃, 𝑥) = 𝑦 (ℓ) (𝜃, 𝑥) 𝜈 . Activation regions are defined by the activation pattern of all the hidden neurons for any input 𝑥. Indeed, denoting the ReLU function by 𝜎, a neuron 𝜈 ∈ 𝐻 is either on20 when 𝑦 𝜈 (𝜃, 𝑥) > 0 or off when 𝑦 𝜈 (𝜃, 𝑥) = 0. For instance, as depicted in the example in Figure 2-7, a neuron 𝜈 belonging to the first hidden layer of the network defines a separating hyperplane Γ 𝜈 in the input space, defined as the set of input points for which the output function of the neuron becomes strictly positive, or equivalently, where the gradient is discontinuous.

Definition 2.2.9 (Bent Separating Hyperplanes). For every 𝜈 ∈ 𝐻, we denote

Γ 𝜈 = { 𝑥 ∈ R 𝑁 0 | 𝑥 ↦ → ∇ 𝜃 𝑦 𝜈 (𝜃, 𝑥) is discontinuous at 𝑥 } .
For admissible parameterizations and for any neuron 𝜈 ∈ 𝑁 1 , Γ 𝜈 is a hyperplane since 𝑦 𝜈 (𝜃) = max (⟨𝑥, 𝑤 •→𝜈 ⟩ + 𝑏 𝜈 , 0)21 and since 𝑤 •→𝜈 ̸ = 0 (admissibility). This corresponds to the two blue lines on the left panel of Figure 234567. For deeper hidden neurons 𝜂 ∈ 𝑁 ℓ , ℓ > 1, Γ 𝜂 is in general the union of (1) some gradient discontinuities due to the lower layers and (2) some gradient discontinuities that are induced by 𝜂. For instance, on Figure 234567, the neuron on the second hidden layer (1) keeps a part of the gradient discontinuities induced by the two neurons of the first hidden layer, namely the lines between the orange and yellow regions, as well as the line between the purple and the yellow regions appearing in the right panel and (2) introduces new gradient discontinuities, namely the orange line depicted on the left panel. The sets Γ 𝜈 are often referred to as "bent separating hyperplane" [START_REF] Hanin | Deep relu networks have surprisingly few activation patterns[END_REF]. The main motivation for this name is that the new gradient discontinuities introduced by one neuron belonging to layer ℓ bend on the bent separating hyperplanes of lower layers ℓ ′ < ℓ as demonstrated by [START_REF] Hanin | Deep relu networks have surprisingly few activation patterns[END_REF]: for instance, on the left panel of Figure 234567, the orange line bends on the two blue lines. However, without any further assumptions, the dimension of Γ 𝜈 might be lower than |𝑁 0 | -122 . Thus, the term "hyperplane" is not necessarily used rigorously in the literature but facilitates greater understanding and we choose to adopt it. Definition 2.2.10 (Activation Regions). Let Γ = ∪ 𝜈∈𝐻 Γ 𝜈 . The activation regions are the connected components of R 𝑁 0 ∖Γ.

Activation regions provide intimate information about each of the network's hidden neurons. One alternative definition proposed by [START_REF] Montúfar | On the number of linear regions of deep neural networks[END_REF] is to assign to each input 𝑥 a vector 𝜌(𝑥) ∈ 2 |𝐻| where, for any hidden neuron 𝜈 ∈ 𝐻, 𝜌(𝑥) 𝜈 = 1 if 𝑦 𝜈 (𝜃, 𝑥) > 0 and 0 otherwise. Then, 𝜌 is piecewise constant and represents the activation pattern of all the hidden neurons for a given input, as depicted in Figure 2-7: for instance, the pattern 𝜌(𝑥) = (0, 1, 1) or "011" means that the neuron 𝜈 1 is off whereas both neurons 𝜈 2 and 𝜂 are on for input 𝑥 (see Figure 2-6 for the architecture and name of the neurons).

Definition 2.2.11 (Activation Regions bis). The activation regions are the areas of the input space where 𝑥 ↦ → 𝜌(𝑥) is constant.

Both definitions coincide (Hanin and Rolnick, 2019, Lemma 2). On the other hand, linear regions are areas of the input space for which the global function implemented by the network 𝑥 ↦ → 𝑅 𝜃 (𝑥), or realization, has distinct locally affine behavior. In the remainder of this manuscript and with a slight abuse of notation, we will interchangeably refer to affine or linear functions. Definition 2.2.12 (Linear regions). For every 𝜈 ∈ 𝐻, we denote

𝐵 = { 𝑥 ∈ R 𝑁 0 | 𝑥 ↦ → ∇ 𝑥 𝑅 𝜃 is discontinuous at 𝑥 } .
Then, the linear regions are the connected components of R 𝑁 0 ∖𝐵.

Activation regions are convex and there are always at least as many activation regions than linear regions (Hanin and Rolnick, 2019, Lemma 3). For instance, the two-hidden-layer network defined in Figure 2-7 has 4 linear regions (see right panel), but 7 activation regions (see left panel). In general, for deeper networks, the number of activation and linear regions grows both exponentially as illustrated in Figure 2345678.

Parameters-Realization Duality

Rolnick and Kording (2019) prove, under certain assumptions detailed in Subsection 2.2.3, that it is possible to recover the network's parameters and architecture from the linear regions 23 . However, it is sometimes not the case. Let us focus on Figure 2-9: Left: example of a dead neuron, 𝜈 2 . Right: example of twin neurons, 𝜈 1 and 𝜈 2 . Squares denote input or output neurons whereas circles denote hidden neurons. Weights are specified on the connections between two neurons and we set all biases to zero (not represented in the figures).

two toy examples, as illustrated in Figure 2-9, to better grasp the duality between the parameters and the realization. We focus on one-hidden-layer networks with

𝑁 0 = {𝜇 1 , 𝜇 2 }, 𝑁 1 = {𝜈 1 , 𝜈 2 }
and 𝑁 2 = {𝜂} and no biases.

• Dead neuron. We set (𝑤

𝜇 1 →𝜈 1 , 𝑤 𝜇 2 →𝜈 1 ) = (1, -1), (𝑤 𝜇 1 →𝜈 2 , 𝑤 𝜇 2 →𝜈 2 ) = (1, 1)
and (𝑤 𝜈 1 →𝜂 , 𝑤 𝜈 2 →𝜂 ) = (1, 0). Then, 𝑅 𝜃 (𝑥) = max(0, 𝑥 𝜇 1 -𝑥 𝜇 2 ) and 𝑅 𝜃 only has two affine regions separated by a line 𝑥 𝜇 1 = 𝑥 𝜇 2 . Hence, the second neuron is dead (see Definition 3.2.1) and does not participate in the realization (since 𝑤 𝜈 2 →𝜂 = 0) and the network cannot be recovered from the realization24 . There are 4 activation regions but the dead neuron prevents them from showing up in the linear regions.

• Twin neurons. We set (𝑤 Note that for the dead neuron example, the parameterization is not admissible since neuron 𝜈 2 is not connected to any output neuron through a path of non-zero weights.

𝜇 1 →𝜈 1 , 𝑤 𝜇 2 →𝜈 1 ) = (1, -1), (𝑤 𝜇 1 →𝜈 2 , 𝑤 𝜇 2 →𝜈 2 ) = (-1, 1) and (𝑤 𝜈 1 →𝜂 , 𝑤 𝜈 2 →𝜂 ) = (1, -1). Then, 𝑅 𝜃 (𝑥) = 𝑥 𝜇 1 -𝑥 𝜇 2 is
Reciprocally, as shown in Chapter 3, if a parameterization 𝜃 is not admissible, then at least one hidden neuron is dead, where a dead neuron is formally defined in 3.2.1). Thus, it is natural to focus our attention to admissible parameterizations in our work. However, the admissibility condition does not exclude all possible degenerated cases as illustrated with the twin neurons example. In the two (degenerate) above cases, the activation regions cannot be recovered from the network realization.Hence, the functional equivalence classes of such examples are more vast than the sole rescalings and permutations (see Chapter 3 for details). We state precise non-degeneracy conditions in Subsection 2.2.3.

Functional Equivalence Classes

In this Subsection, we review related work aiming at characterizing functional equivalence classes under some precise conditions, sometimes called non-degeneracy or irreducibility conditions. We enumerate the known results along with their scope, the formulation of their assumptions and their limitations. Note that some results may not encompass the ReLU networks since they were proved before the recent introduction of the ReLU. In this case, other activation functions might necessitate sign flips instead of rescalings as defined in Subsection 2.2.1. The main difficulty is to properly identify the non-degeneracy conditions under which it is possible to easily describe the equivalence classes as rescalings (or sign flips) and permutations.

Definition 2.2.13. Let 𝐺 be a network architecture valued with the parameterizations 𝜃 and 𝜃 ′ . Then, 𝜃 and 𝜃 ′ are said to be functionally equivalent if, for all 𝑥 ∈ R 𝑁 0 , 𝑅 𝜃 (𝑥) = 𝑅 𝜃 ′ (𝑥). We denote by ∼ 𝐹 the resulting equivalence relation.

In this definition, we assume that 𝜃 and 𝜃 ′ share a common network architecture 𝐺. A more general definition would allow for two different architectures 𝐺 and 𝐺 ′ on the only conditions that 𝐺 and 𝐺 ′ share the same set of input neurons 𝑁 0 and output neurons26 . As explained below, for the one-hidden-layer case, under proper irreducibility conditions, two functionally equivalent networks automatically share the same architecture: equivalently, two functionally equivalent one-hidden-layer networks valued with irreducible parameterizations 𝜃 and 𝜃 ′ have the same number of hidden neurons 𝑁 1 . The question is more complex for deeper networks.

One Hidden Layer

We detail three successive approaches for the one-hidden-layer case that were designed in the early 90's, for activation functions satisfying various conditions27 . These three results do not encompass the ReLU case as shown in

𝑅 𝜃 (𝑥) = ∑ 𝜈∈𝑁 1 𝑤 𝜈→𝜂 𝜎 (⟨𝑤 •→𝜈 , 𝑥⟩ + 𝑏 𝜈 ) + 𝑏 𝜂 .
All theorems below consider two parameterizations 𝜃 and 𝜃 ′ on two architectures 𝐺 and 𝐺 ′ having the same input and output neurons but a priori not the same number of hidden neurons. As summarized in Table 2.1, the results specify (1) a family of activation functions under study; (2) irreducibility conditions on 𝜃 and 𝜃 ′ ; such that functional equivalence implies that the number of hidden neurons is the same and that 𝜃 ′ and 𝜃 are permutation sign-flip equivalent. Sussmann (1992) considers only the tanh activation function, and specifies the irreducibility condition in terms of dead and twin neurons (See Subsection 2.2.2). Informally, a dead neuron does not participate to the output function of the network (for instance when all its incoming or outgoing weights are set to zero), whereas twin neurons can be collapsed to a single node without altering the input-output map. We study these notions more formally in Chapter 3. [START_REF] Albertini | Uniqueness of weights for neural networks[END_REF] consider any odd activation function 𝜎 that satisfies the independence property: for every positive integer 𝑝, for every 𝑐 1 , . . . , 𝑐 𝑝 ∈ R * and every 𝛾 1 , . . . , 𝛾 𝑝 ∈ R such that (𝑐 𝑖 , 𝛾 𝑖 ) ̸ = ±(𝑐 𝑗 , 𝛾 𝑗 ) for all 𝑖 ̸ = 𝑗 the functions

𝑥 ↦ → 1, 𝑥 ↦ → 𝜎(𝑐 1 𝑥 + 𝛾 1 ), . . . , 𝑥 ↦ → 𝜎(𝑐 𝑝 𝑥 + 𝛾 𝑝 )
(2.10) are linearly independent. Following earlier work by [START_REF] Kůrková | Functionally equivalent feedforward neural networks[END_REF], [START_REF] Kainen | Uniqueness of network parameterizations and faster learning[END_REF]

, 𝜈 2 ∈ 𝑁 1 , 𝜈 1 ̸ = 𝜈 2 , 𝑤 •→𝜈 1 ̸ = 𝑤 •→𝜈 2
Table 2.1: Specific functional equivalence results for one single hidden layer. All statements read as follows: for the family of activation functions, under the irreducibility conditions, the input-output map of a network uniquely determines its architecture and its parameters up to permutations and sign flips (see Subsection 2.2.1).

non-polynomial rational functions.

The irreducibility conditions summarized in Table 2.1 may differ in appearance from the original notations and statements on the papers. However, we have simply reformulated the assumptions to make them more consistent and readable and refer to the original papers for further details.

For the sake of clarity, own stand-alone proof for single-hidden-layer ReLU networks found independently, is available in Chapter 3, Proposition 3.2.1. The irreducibility condition reads: there are no dead hidden neurons and no twin hidden neurons as defined in Definitions 3.2.1 and 3.2.2. Note that this proof includes the case of multiple output neurons, i.e. |𝑁 2 | > 1, unlike the results detailed in Table 2.1.

Many Hidden Layers

Unfortunately, the proof techniques used in the single-hidden-layer case rely on geometrical arguments and are not readily adaptable for deeper networks. We detail below the known results and summarize them in 2.2: Specific functional equivalence results for multiple hidden layers. All statements read as follows: for the family of activation functions, under the irreducibility conditions, the input-output map of a network uniquely determines its architecture (including depth) and its parameters up to symmetries. The symmetries depend on the activation function and are specified in the same column in parenthesis. Symmetries always include permutations and include either rescalings or sign flips (see 2.2.1). † The result of [START_REF] Phuong | Functional vs. parametric equivalence of relu networks[END_REF] writes as follows: given a bounded nonempty domain 𝑍 ⊂ R 𝑁 0 , for eligible architectures (with non-increasing layer widths), there exists a parameterization 𝜃 such that the knowledge of the realization 𝑅 𝜃 on 𝑍 determines the network's parameters uniquely up to permutations and scalings among general and transparent networks.

The first known exact result for networks with many hidden layers is due to [START_REF] Fefferman | Reconstructing a neural net from its output[END_REF] in the case of the tanh non-linearity. Indeed, before the introduction of the ReLU non-linearity [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], analogies to the human nervous system suggested to use a sigmoid-like activation function 𝜎(𝑥) asymptotic to constants as 𝑥 → ±∞. The non-degeneracy conditions are specified as follows:

(F.1) Non-nullity. 𝑏 𝜈 ̸ = 0 for all 𝜈 ∈ 𝐻 and 𝑤 𝑒 ̸ = 0 for all 𝑒 ∈ 𝐸.

(F.2) Non-equality. (i) For all 𝜈 ̸ = 𝜈 ′ ∈ 𝐻, |𝑏 𝜈 | ̸ = |𝑏 𝜈 ′ |.
(ii) For all ℓ ∈ 1, 𝐿 , for all 𝜈 ̸ = 𝜈 ′ ∈ 𝑁 ℓ , for all 𝜇 ∈ 𝑁 ℓ-1 , the ratio 𝑤 𝜇→𝜈 /𝑤 𝜇→𝜈 ′ is not equal to any fraction of the form 𝑝/𝑞, with 𝑝, 𝑞 integers and 1

≤ 𝑞 ≤ 100|𝑁 ℓ | 2 .
The statement reads as follows: under the two conditions above, perfect knowledge of the input-output map of a network uniquely specifies both the architecture (including depth) and the parameters, up to permutations and sign flips -we consider sign flips since the non-linearity is the hyperbolic tangent, see Definition 2.2.8. The idea of the proof is to reduce to the scalar case where |𝑁 0 | = |𝑁 𝐿 | = 1 and to continue the realization 𝑓 (𝑥) = 𝑅 𝜃 (𝑥) analytically to a function 𝑓 of a single, complex variable. Then, the qualitative geometry of the poles of 𝑓 determines the network architecture and the asymptotics of 𝑓 near its singularities determines the weights. The proof is not readily adaptable to ReLU networks since it heavily relies on the disposition of the poles of the meromorphic function 𝑧 ↦ → tanh(𝑧/2)29 . [START_REF] Rolnick | Reverse-engineering deep relu networks[END_REF] propose to reverse-engineer deep ReLU networks by reading both the architecture of the network (including depth) and its parameters from the sole knowledge of the linear regions and each affine function implemented in these regions by the network (see Subsection 2.2.2 for definitions). The authors propose a constructive algorithm that samples network realizations for carefully chosen input points to deduce the architecture of the network and its parameters, up to rescalings and permutations. The authors make two crucial assumptions: (R.1) Linear Regions Assumption. "Each [activation] region represents a maximal connected component of input space on which the piecewise linear function 𝑥 ↦ → 𝑅 𝜃 (𝑥) is given by a single linear function" (see Definition 2.2.10).

(R.2) For any edge 𝑒 = 𝜇 → 𝜈, Γ 𝜇 ∩ Γ 𝜈 ̸ = ∅, i.e. two connected hidden neurons have bent separating hyperplanes that intersect, see Definition 2.2.9.

(R.

3) The parameters 𝜃 do not belong to a certain measure-zero set.

Let us discuss assumption R.1 further. Recall that activation regions (Definition 2.2.10) are the areas of the input space that correspond to constant activation patterns of all the hidden neurons, whereas linear regions (Definition 2.2.12) correspond to the maximally connected areas of the input space for which the realization 𝑅 𝜃 is an affine function. Assumption R.1 states that activation regions and linear regions coincide, which is not always the case (see Subsection 2.2.2). For instance, the network considered in Figure 2-7 has 7 activations regions but only 4 linear regions30 , and does not satisfy R.1. The authors prove that their algorithm terminates31 except for a measure-zero32 set of networks as stated in assumption R.3. In other words, under their assumptions, two functionally equivalent networks are permutation-rescaling equivalent. While their algorithm is constructive, the two assumptions are made on the linear regions and not on the network's parameters themselves, which might hide the underlying causes of such situations. Moreover, some precisions could have been made about the measure-zero set of network: which probability measure is used, and what are canonical examples of networks in this set.

Similarly, [START_REF] Phuong | Functional vs. parametric equivalence of relu networks[END_REF] propose to use linear regions to identify the network's parameters. They make the following assumptions that we cite below.

(P.1) General Network.

(i) For any 𝜈 ∈ 𝐻, the local extrema of 𝑥 ↦ → 𝑦 𝜈 (𝜃, 𝑥) do not have value zero.

(ii) For all 𝑘, ℓ such that 0 < 𝑘 ≤ ℓ < 𝐿 and all diagonal matrices (𝐼 (𝑘) , . . . , 𝐼 (ℓ) )

with entries in {0, 1} and with

𝐼 (ℓ ′ ) ∈ R |𝑁 ℓ ′ | , rank ( 𝑊 (𝑘) 𝐼 (𝑘) • • • 𝑊 (ℓ) 𝐼 (ℓ)
)

= min

( |𝑁 𝑘-1 |, rank ( 𝐼 (𝑘)
)

, . . . , rank

( 𝐼 (ℓ)
)) .

(iii) For all 𝜇 ̸ = 𝜈 ∈ 𝐻, any linear regions 𝐴 1 ̸ = 𝐴 2 of 𝑅 𝜃 , the linear functions implemented by neuron 𝜈 on 𝐴 1 and neuron 𝜇 on 𝐴 2 are not multiple of each other.

(P.2) Transparent Network. For any input 𝑥, there exists at least one hidden neuron 𝜈 such that 𝑦 𝜈 (𝜃, 𝑥) > 0.

(P.3) Eligible architecture. The network architecture 𝐺 satisfies

|𝑁 𝐿 | = 1 and |𝑁 0 | ≥ |𝑁 1 | ≥ • • • ≥ |𝑁 𝐿-1 | ≥ 2.
Assumption P.3 is quite restrictive in terms of considered networks. The authors prove a much weaker result than the previous ones that states as follows: given a bounded non-empty domain 𝑍 ⊂ R 𝑁 0 , for eligible architectures (with non-increasing layer widths), there exists a parameterization 𝜃 such that the knowledge of the realization 𝑅 𝜃 on 𝑍 determines the network's parameters uniquely up to permutations and scalings among general and transparent networks. Thus, given any parameterization 𝜃, the result does not state that its parameters are uniquely determined up to permutations and scalings, even if 𝜃 is general with an eligible architecture. The assumptions however allow the authors to prove that almost all ReLU networks are general: "In other words, a sufficient condition for a [random] ReLU network to be general with probability one is that its weights are sampled from a distribution with a density". The authors consider only the scalar output case and do not formulate theorems for non-eligible architectures.

Relaxed Equivalence Classes

For many applications, for instance image classification, the quantity of interest it not the function implemented by the network 𝑅 𝜃 or the loss, but rather the classification decision. In the ImageNet setup [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], there are 1000 classes, thus 𝑁 𝐿 = 1000 and the predicted class 𝑐 for an input 𝑥 is

𝑐 = argmax 𝜈∈𝑁 𝐿 (𝑅 𝜃 (𝑥) 𝜈 ).
Thus, various parameterizations 𝜃 and 𝜃 ′ of the same network might not be functionally equivalent, in the sense that 𝑅 𝜃 = 𝑅 ′ 𝜃 , but may output exactly the same classification decisions for a given set of inputs, usually if their realizations are close enough. From this point of view, the strict concept of functional equivalence classes defined in Subsection 2.2.3 might be relaxed to preserve -approximately or exactlya higher-level quantity of interest, such as the loss or the classification error, for the whole input space R 𝑁 0 or for a given distribution or set of test inputs.

In this section, we focus on the concept of inverse stability [START_REF] Petersen | Topological properties of the set of functions generated by neural networks of fixed size[END_REF][START_REF] Berner | How degenerate is the parametrization of neural networks with the relu activation function? NeurIPS[END_REF]. Consider a fixed ReLU architecture valued with two parameterizations 𝜃 and 𝜃 ′ and the following question.

"If the realizations 𝑅 𝜃 and 𝑅 𝜃 ′ are close, are the parameterizations 𝜃 and 𝜃 ′ close?" This question is ill-posed and fundamentally connected to understanding the redundancies in the parameterizations of neural networks, precisely because of the rescaling and permutation operations defined in Subsection 2.2.1 that preserve the function implemented by the network. For instance, assume that 𝑅 𝜃 and 𝑅 𝜃 ′ , as well as 𝜃 and 𝜃 ′ , are close. We might rescale 𝜃 ′ into 𝜃 ′′ such that 𝑅 𝜃 ′ = 𝑅 𝜃 ′′ and such that 𝜃 ′ and 𝜃 ′′ are arbitrarily far from each other. More precisely, following Definition 2.2.3, consider one single hidden neuron 𝜈 and define, for any 𝜆 𝜈 > 0, 𝑠 = 𝑠 𝜈,𝜆𝜈 and 𝜃 ′′ = 𝑠(𝜃 ′ ). Further assume that the biases are all set to zero. Then,

‖𝜃 ′′ -𝜃 ′ ‖ 2 2 = ∑ 𝜇∈prev(𝜈) (𝜆 𝜈 𝑤 𝜇→𝜈 -𝑤 𝜇→𝜈 ) 2 + ∑ 𝜂∈next(𝜈) (𝑤 𝜈→𝜂 /𝜆 𝜈 -𝑤 𝜈→𝜂 ) 2 = (𝜆 𝜈 -1) 2 ∑ 𝜇∈prev(𝜈) 𝑤 2 𝜇→𝜈 + (1/𝜆 𝜈 -1) 2 ∑ 𝜂∈next(𝜈) 𝑤 2 𝜈→𝜂 .
Hence, as 𝜆 𝜈 → +∞, ‖𝜃 ′′ -𝜃 ′ ‖2 2 → +∞ and we still have 𝑅 𝜃 ′ = 𝑅 𝜃 ′′ with 𝜃 ′ and 𝜃 ′′ far from each other. Since 𝑅 𝜃 and 𝑅 𝜃 ′ are close, 𝑅 𝜃 and 𝑅 𝜃 ′′ are also close. Finally, since 𝜃 and 𝜃 ′ are close, 𝜃 and 𝜃 ′′ are far from each other. Thus, [START_REF] Berner | How degenerate is the parametrization of neural networks with the relu activation function? NeurIPS[END_REF] propose to reformulate the question as follows.

"If the realizations 𝑅 𝜃 and 𝑅 𝜃 ′ are close, can we find a parameterization 𝜃 ′′ that is permutation-rescaling equivalent to 𝜃 ′ such that 𝜃 and 𝜃 ′′ close?"

This question is addressed by [START_REF] Berner | How degenerate is the parametrization of neural networks with the relu activation function? NeurIPS[END_REF] in the one-hidden-layer case with no biases and scalar output. They exclude degenerate cases as follows.

(B.1) For all 𝜈 1 , 𝜈 2 ∈ 𝑁 1 such that ‖𝑤 •→𝜈 1 ‖ ∞ > 0 and ‖𝑤 •→𝜈 2 ‖ ∞ > 0, 𝑤 •→𝜈 1 ‖𝑤 •→𝜈 1 ‖ ∞ ̸ = 𝑤 •→𝜈 2 ‖𝑤 •→𝜈 2 ‖ ∞ . (B.2) For all 𝜈 ∈ 𝑁 1 , ‖𝑤 𝜈→• ‖ ∞ = ‖𝑤 •→𝜈 ‖ ∞ . (B.
‖𝜃 ′′ -𝜃‖ ∞ ≤ 4|𝑅 𝜃 ′′ -𝑅 𝜃 | 1 2 𝑊 1,∞ .
Note that the distance between realizations is controlled with a Sobolev semi-norm. Here, we are interested in the case 𝛿 = 𝜂 = 0, i.e. local identifiability. In Section 6, the authors provide sufficient and necessary conditions for local identifiability by studying complex algebraic varieties leveraging the Segre embedding of the ℎ 𝑖 . We refer to the paper for further details.

• Malgouyres (2020) also consider deep structured linear networks, without biases, under some sparsity constraints on the parameters ℎ 𝑖 ∈ R 𝑆 . The authors formalize the stability guarantees slightly differently than [START_REF] Malgouyres | Multilinear compressive sensing and an application to convolutional linear networks[END_REF], as stated in the Informal Theorem 1 in the paper. Let 𝑑 be a metric taking into account the inter-layer rescaling of the weights. Define the empirical risk on a training set (𝑥 𝑖 , 𝑦 𝑖 ) 𝑖=1..𝑛 for a parameterization 𝜃 as

𝐸(𝜃) = ∑ 𝑖 ‖𝑦 𝑖 -𝑅 𝜃 (𝑥 𝑖 )‖.
Then, the stability property writes: there exists 𝐶 ≥ 0 such that, for 𝜂 sufficiently small and for any 𝜃, 𝜃 ′ such that 𝐸(𝜃) ≤ 𝜂 and 𝐸(𝜃 ′ ) ≤ 𝜂, we have 𝑑(𝜃, 𝜃 ′ ) ≤ 𝐶𝜂. In this work, the authors provide sharp conditions on the network architecture as well as the training set such that the parameters obtained after optimization in the training set are uniquely defined, up to layer-wise rescalings, again instead of neuron-wise rescalings.

Applications

As seen in the previous Subsection 2.2.3 for any activation function but more extensively for ReLU networks, functional equivalence classes are described by the sole permutation and rescaling operations, except for degenerate cases. In practice, researchers implicitly assume that the degenerate cases appear rarely, or even for a measure-zero set of networks ( [START_REF] Rolnick | Reverse-engineering deep relu networks[END_REF]). Moreover, since permutations are discrete operations33 , the authors mainly focus on the rescaling operations to describe functional equivalence classes. Since various representants of the same equivalence classes may react differently to optimization [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF] or scalar quantization [START_REF] Nagel | Data-free quantization through weight equalization and bias correction[END_REF], the main difficulty lies in identifying a possible canonical representant, if any, for each equivalence class given the task at hand. In the following, we say that a network is balanced if all the weights have roughly the same magnitude, and unbalanced if this not the case, for instance when rescaling with large coefficients as explained in Subsection 2.2.4. An illustration of balanced and unbalanced networks implementing the same realization is given in Figure 2-10.

Quotient SGD

As observed by [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF], the balancedness of the network plays an important role in the optimization process. More precisely, given an architecture 𝐺, we compare a parameterization 𝜃 initialized with standard methods (as described in Subsection 2.1.3) and one rescaled, unbalanced parameterization 𝜃 ′ that yields the same realization as 𝜃, as illustrated in Figure 2-10. It has been observed that, when initialized in an unbalanced fashion, neural networks are slower to train and can even converge to a poor local minimum 34 .

Hence, [START_REF] Meng | 𝒢-SGD: Optimizing relu neural networks in its positively scale-invariant space[END_REF] propose to study the optimization problem over the space of realizations of neural networks rather than the space of parameters. The practical idea is to re-balance the network with a rescaling-equivalent network after each SGD step: this operation does not change the realization implemented by the network but will arguably change the future gradients and the learning trajectory 35 . More formally, this two-step optimization process amounts to quotient the parameter space with the rescaling equivalence relation ∼ 𝑆 , implicitly assimilated to the functional equivalence relation ∼ 𝐹 , where both notations are defined in Subsection 2.2.1. This line of work is related to our contribution in Chapter 4 where we derive a balancing algorithm that provably converges to a canonical representant for linear and convolutional networks and leverage it to improve the training. As stated in Chapter 4, the proposed algorithm -hence the obtained canonical representant -is justified by its practical efficiency. However, we make no theoretical claim supporting the fact that the proposed algorithm leads to the right canonical representant. Indeed, there may be alternative definitions leading to distinct canonical representants and we still miss a criterion to define "how good" a representant is.

Weight Equalization

Inspired by our contribution in Chapter 4, [START_REF] Nagel | Data-free quantization through weight equalization and bias correction[END_REF] and [START_REF] Meller | Same, same but different -recovering neural network quantization error through weight factorization[END_REF] concurrently use our Equi-Normalization algorithm to balance networks and therefore make them more amenable to 8-bit scalar quantization (see Subsection 2.3.5 for details about quantization). [START_REF] Nagel | Data-free quantization through weight equalization and bias correction[END_REF] conjecture that the performance of trained models after quantization can be improved by adjusting the weights of each channel such that their ranges are more similar. As the authors empirically demonstrate, the quantization error induced on a single layer by traditional methods such as Quantization-Aware Training (QAT, [START_REF] Krishnamoorthi | Quantizing deep convolutional networks for efficient inference: A whitepaper[END_REF]) may induce a drift in the distribution of the activations of successive layers with potential damaging effects on the network's accuracy. Such methods are discussed in Subsection 2.3.5.

Compression of Deep Learning Models

Deep Learning models are able to perform multiple tasks such as image recognition, language understanding or speech enhancement thanks to a very active research community. Productionizing such over-parameterized -hence heavy -models, either on the server side or on the mobile side, poses challenges in terms of latency, memory and energy. In this Section, we detail the main compression techniques to produce models with fast inference speed, small memory footprint and energy efficiency. One of the most straightforward methods to compress a neural network is to prune or delete some of its parts (Subsection 2.3.1). More structured sparsity leads to the design of Structured Efficient Layers detailed in Subsection 2.3.2. This work in turns conducts the research community to construct or search for new efficient architectures (Subsection 2.3.3). Networks with these architectures are either learnt from scratch or using a more advanced teacher network with distillation techniques (Subsection 2.3.4). Moreover, reducing the precision at which to store single weights or subvectors leads to the fields of scalar and vector quantization (Subsection 2.3.5). Finally, we discuss hardware and metrics considerations in Subsection 2.3.6.

Pruning and Sparsity

Pruning amounts to remove some part of a network -such as a connection or a neuron -according to some importance criterion [START_REF] Lecun | Optimal brain damage[END_REF][START_REF] Hassibi | Second order derivatives for network pruning: Optimal brain surgeon[END_REF]. Generally, researchers start from a pretrained network and iteratively prune and finetune the network until the desired size/accuracy tradeoff is reached.

Importance criteria

The importance criteria are designed to quantify the effect of deleting a given part of the network on its accuracy. Then, the parts with the lowest importance are removed (or equivalently, freezed to zero 36 ). For the sake of clarity, let us write the most common importance criteria when pruning connections (i.e., individual weights) in a network -sometimes called unstructured pruning. These methods generalize to larger structures such as neurons, filters or layers as explained in the references below.

Let us denote the weights of the network by 𝑤 = (𝑤 𝑒 ) 𝑒∈𝐸 , where 𝐸 is the set of edges or connections of the network's architecture. We aim at defining an importance criterion 𝐶 : 𝐸 ↦ → R that will be used to rank the weights. One of the most straightforward implementation of 𝐶 is to consider the magnitude of the weights:

𝐶(𝑒) = |𝑤 𝑒 |.
(

2.11)

There is a large line of work considering weight magnitude pruning [START_REF] Han | Learning both weights and connections for efficient neural network[END_REF]Li et al., 2016;[START_REF] Gale | The state of sparsity in deep neural networks[END_REF] with extensions to the 𝐿 𝑝 norm when considering larger structures such as filters37 . The intuition behind this criterion is that large weights (in absolute value) have a large influence on the loss function ℒ whereas small weights have a negligible influence38 .

In line with the Optimal Brain Damage approach of [START_REF] Lecun | Optimal brain damage[END_REF], it is possible to construct a local model of the loss function ℒ to predict the effect of perturbing the parameters by 𝛿𝑤. Denoting ∇ℒ(𝑤) the gradient of ℒ with respect to 𝑤 and 𝐻(𝑤) the Hessian, the Taylor approximation writes

𝛿ℒ = ∑ 𝑒 ∇ℒ(𝑤) 𝑒 𝛿𝑤 𝑒 + 1 2 ∑ 𝑒 𝐻(𝑤) 𝑒𝑒 𝛿𝑤 2 𝑒 + 1 2 ∑ 𝑒̸ =𝑒 ′ 𝐻(𝑤) 𝑒𝑒 ′ 𝛿𝑤 𝑒 𝛿𝑤 𝑒 ′ + 𝑂(‖𝛿𝑤‖ 3 ).
(2.12)

Pruning a single weight 𝑤 𝑒 is equivalent to applying a perturbation 𝛿𝑤 to 𝑤 such that

𝛿𝑤 𝑒 ′ = -𝑤 𝑒 ′ if 𝑒 ′ =
𝑒 and 0 otherwise. Let us examine how to derive first and second order importance criteria.

• First order. If we neglect all the second-order terms in Equation (2.12), following [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient transfer learning[END_REF]; Yu et al. (2018b), we obtain 𝛿ℒ = ∑ 𝑒 ∇ℒ(𝑤) 𝑒 𝛿𝑤 𝑒 . Then, during training, we wish to prune a single weight 𝑤 𝑒 that will result in the minimum absolute loss variation |𝛿𝐿|. Using perturbations defined earlier, we are left to find 𝑒 such that |∇ℒ(𝑤) 𝑒 𝑤 𝑒 | is minimum. This leads to the criterion

𝒞(𝑒) = |∇ℒ(𝑤) 𝑒 𝑤 𝑒 | .
(2.13)

Pruning with this criterion is sometimes referred to as gradient magnitude pruning. See Figure 2-11 for a comparison.

• Second order. Following [START_REF] Lecun | Optimal brain damage[END_REF], we assume that the training has converged at a local minimum, thus ∇ℒ(𝑤) = 0. We also assume that the Hessian is diagonal, thus we neglect the cross-error terms of the form

𝐻(𝑤) 𝑒𝑒 ′ 𝛿𝑤 𝑒 𝛿𝑤 𝑒 ′ for 𝑒 ̸ = 𝑒 ′ .
Based on these two assumptions and the fact that the Hessian is positive semidefinite at a local minimum, we deduce that 𝐻(𝑤) 𝑒𝑒 ≥ 0. Then Equation (2.12) boils down to 𝛿ℒ = 1 2 ∑ 𝑒 𝐻(𝑤) 𝑒𝑒 𝛿𝑤 2 𝑒 , and we wish to prune a single weight 𝑤 𝑒 that will result in the minimum absolute loss increase 𝛿𝐿, which leads to the criterion

𝒞(𝑒) = 𝐻(𝑤) 𝑒𝑒 𝑤 2 𝑒 .
(2.14)

However, computing second-order information involves additional code and relies on some approximations on the Hessian39 .

Such first-and second-order criteria are sometimes referred to as weight saliencies. In contrast to the weight magnitude, these criteria are data-dependent, and the gradients or the (approximated) Hessian are generally computed on a single batch of training data. In practice, we prune a proportion 𝑝 of the weights at the same time, neglecting the influence of the deletion of one parameter to the saliency of other parameters. Note that biases are generally not pruned, since they represent a small proportion of the networks' parameters, although the same reasoning could apply. Finally, there is an impressive amount of importance criteria in the literature, for instance based in filter statistics, entropy or mutual information. Moreover, the scale at which to apply these criteria -single layer or entire network -may vary. See For extensive references, we refer the reader to the surveys by [START_REF] Liu | Rethinking the value of network pruning[END_REF] or [START_REF] Blalock | What is the state of neural network pruning? MLSys[END_REF]. Finally, the recent work by [START_REF] Frankle | The lottery ticket hypothesis: Training pruned neural networks[END_REF] constitute an interesting direction to extract performant subnetworks40 that train efficiently [START_REF] Caron | Pruning convolutional neural networks with self-supervision[END_REF]. 2020). Weight and Gradient refer respectively to weight magnitude pruning and gradient magnitude pruning. If these criteria are applied per layer, the method is prefixed with Layer. If the criteria are applied to the whole network, the method is prefixed with Global.

Pruning in practice

Pruning is a very common technique for network compression and is for instance implemented directly in PyTorch 41 . Users can perform random or magnitude pruning or even define their own criterion to prune connections, neurons or channels. On a side note, a technique for pruning 42 a given set of entire layers at test time called Stochastic Depth was developed and by [START_REF] Huang | Deep networks with stochastic depth[END_REF] for Vision architectures and extended and modified by [START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF] in Natural Language Processing tasks with Transformers [START_REF] Vaswani | Attention is all you need[END_REF].

Pruning standard networks like a ResNet-18 (He et al., 2015a) on ImageNet with pruning baselines in Equations (2.11) and (2.13) generally leads to a 2 to 4× compression ratio at the cost of a few percentage point in test top-1 accuracy, as depicted in Figure 2-11. We observe a set of empirical Pareto Curves: the more we compress the network, the more the accuracy decreases. Note that considering the criteria (weight magnitude or gradient magnitude) for the whole network 43 yields better accuracy than considering them per layer 44 .

41 https://pytorch.org/tutorials/intermediate/pruning_tutorial. 42 More precisely, a "pruned" layer ℓ is replaced by the identity layer so that the output of the previous layer 𝑦 (ℓ-1) is directly fed to layer ℓ + 1.

43 Rank all the weights 𝑤 𝑒 , 𝑒 ∈ 𝐸 and prune them according to the ranking. 44 Rank the weights 𝑤 𝜇→𝜈 , ℓ(𝜈) = ℓ per-layer and prune them according to the per-layer ranking.

Structured Efficient Layers

In this Subsection, we seek to replace a dense weight matrix by a matrix or a product or matrices that have fewer parameters while being faster at inference time. The main constraint is that the parameters must be learnt -we do not seek to approximate a dense trained matrix but rather to directly learn the structured matrix, so the factorization must be compatible with back-propagation 45 . We briefly review the main ideas below and point the interested reader to the survey by [START_REF] Cheng | A survey of model compression and acceleration for deep neural networks[END_REF]. In this subsection we restrict ourselves to the 2𝐷 case and seek to approximate a weight matrix 𝑊 ∈ R 𝑛×𝑛 for simplicity.

Low Rank

One of the most common ways to approximate 𝑊 while saving parameters is to create a bottleneck in the network by writing 𝑊 = 𝑈 𝑉 where 𝑈 ∈ R 𝑛×𝑝 and 𝑉 ∈ R 𝑝×𝑛 where 𝑝 is smaller than 𝑛. This is equivalent to replacing the fully connected 𝑛 × 𝑛 layer by two stacked fully-connected layers of dimensions 𝑛 × 𝑝 and 𝑝 × 𝑛, respectively. The compression ratio is 2𝑝/𝑛. This layer is generally a strong baseline, as the resulting network will be forced to find a compressed representation of the information, which is the idea at the core of the auto-encoders [START_REF] Bengio | Learning deep architectures for ai. Found[END_REF]. The Low Rank (or bottleneck) paradigm was extremely fruitful. For instance, generalizations of the lowrank decomposition such as Tensor Train [START_REF] Oseledets | Tensor-train decomposition[END_REF] or T-Net [START_REF] Kossaifi | T-Net: Parametrizing fully convolutional nets with a single high-order tensor[END_REF]; some Transformer architectures using a low-rank approximation or the selfattention mechanism such as the Linformer [START_REF] Wang | Linformer: Self-attention with linear complexity[END_REF], and approximations of the softmax 46 [START_REF] Grave | Efficient softmax approximation for gpus[END_REF] are all inspired by the low-rank paradigm.

Structured Sparse

Using a random sparse matrix instead of a dense one, with a proportion 𝛼 of non-zero coefficients amounts to prune a proportion 1 -𝛼 of the weights before the training [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient transfer learning[END_REF]. For such matrices, sparse calculations rely on optimized kernels 47 . However, more structured operations may be faster. For instance, a structured layer with a block-diagonal matrix, a permutation of the features and another 45 More precisely, the structured layers could be initialized with an approximation of the dense matrix instead of being randomly initialized, and then finetuned. In practice, finetuning is necessary to get good performance, therefore authors generally skip this initialization step (when it is computationally tractable). 46 The softmax function 𝑓 : R 𝑑 ↦ → R 𝑑 is defined as 𝑓 (𝑥) 𝑗 = 𝑒 𝑥 𝑗 ∑ 𝑖 𝑒 𝑥 𝑖 . 47 Such as the cuSPARSE kernel on Nvidia's GPUs: https://developer.nvidia.com/cusparse. block-diagonal matrix. This parameterization relies on the batch-multiplication operation. We define a block-diagonal matrix with 𝑘 blocks 𝑀 𝑖 of size 𝑝 × 𝑝 as

𝐵 = ⎛ ⎜ ⎜ ⎜ ⎝ 𝑀 1 • • • 0 . . . . . . . . . 0 • • • 𝑀 𝑘 ⎞ ⎟ ⎟ ⎟ ⎠ .
Then, the structured layer writes 𝑊 = 𝐵 1 𝑃 𝐵 2 where 𝐵 1 and 𝐵 2 are learnt blockdiagonal matrices and 𝑃 is a permutation matrix (which is not learnt but fixed). The idea is to combine local linear interactions in 𝐵 1 and 𝐵 2 interleaved by a permutation to add some diversity. The compression ratio is 2𝑘𝑝 2 /𝑛 2 . Zhang et al. (2017b) use this idea to design an efficient network based on group convolutions -the convolutional equivalent of the block-diagonal matrix 𝐵 -and permutations called the ShuffleNet, depicted in Figure 2-12.

Fourier and Hadamard

A few methods rely on the discrete Fourier or Hadamard transforms, such as Fast-Food (Le et al., 2014), ACDC [START_REF] Moczulski | ACDC: A structured efficient linear layer[END_REF] and their variants [START_REF] Arjovsky | Unitary evolution recurrent neural networks[END_REF][START_REF] Choromanska | Binary embeddings with structured hashed projections[END_REF]. these methods rely on the Fast Fourier or Hadamard Transform and its 𝑂(𝑛 log 𝑛) implementation using the Cooley-Tukey algorithm [START_REF] Cooley | Historical notes on the fast fourier transform[END_REF].

Let us give some insights explaining why such transforms are used, besides their computational advantage. We rely on the theory of Fast Random Projections developed by [START_REF] Ailon | Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform[END_REF]. The goal of this approach is to perform random projections that are computationally cheap in time and space, that approximately preserve the metric information and that are not learnt. In a random projection, the basic operation takes the form 𝑦 = 𝑊 𝑥, where 𝑊 is a random matrix. As stated by the famous Johnson-Leidenstrauss Lemma [START_REF] Johnson | Extensions of lipschitz mappings into a hilbert space[END_REF], the obtained embeddings approximately preserve metric information48 when 𝑊 is a Gaussian matrix (a matrix whose entries are drawn independently from 𝒩 (0, 1)). [START_REF] Ailon | Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform[END_REF] introduced an alternative approach that mimics the properties of random Gaussian matrices but that can be stored very efficiently. In particular, they proposed the PHD transform: 𝑦 = 𝑃 𝐻𝐷𝑥 where 𝑃 is a sparse random matrix with Gaussian entries, 𝐻 is the Hadamard matrix and 𝐷 a diagonal matrix with entries in {-1, +1} drawn independently with probability 1/2.

The transformations that we will detail now (ACDC and FastFood) can be seen as an alternative to this fast random projection where some of the parameters are learnt. The idea is to keep the approximate metric preservation properties while letting the network adapt to this particular layer. Denoting diagonal matrices of learnt parameters with the letter 𝐷, fixed permutations matrices with 𝑃 , the Hadamard transform with 𝐻 and the Fourier transform with 𝐹 , the FastFood transform writes

𝐷 1 𝐻𝐷 2 𝑃 𝐻𝐷 3 whereas the ACDC transform corresponds to 𝐷 1 𝐹 𝐷 2 𝐹 -1 .

Circulant Matrices and their Variants

We now review some classical learnt structured matrices whose storage complexity is linear [START_REF] Sindhwani | Structured transforms for small-footprint deep learning[END_REF]. The circulant matrix of size 𝑛 is defined with (𝑢 0 , . . . , 𝑢 𝑛-1 ) ∈ R 𝑛 and this vector is successively shifted to generate the next columns. Similar matrices are the Vandermonde and Cauchy matrices. The structure of these matrices can be exploited for faster linear algebraic operations such as matrix-vector multiplication, inversion and factorization. In particular, the matrixvector multiplication with circulant matrix is computable in in 𝑂(𝑛 log 𝑛) time using the Fourier Transform. Vandermonde and Cauchy matrix-vector operations take 𝑂(𝑛 log 2 𝑛) time [START_REF] Sindhwani | Structured transforms for small-footprint deep learning[END_REF]. Note that there exist infinite possibilities of designing structured matrices and combining them to obtain structured layers (see [START_REF] Choromanska | Binary embeddings with structured hashed projections[END_REF]) for further examples. [START_REF] Jose | Kronecker recurrent units[END_REF] consider parameterizing the matrix 𝑊 as a Kronecker product of 𝑘 learnt matrices called Kronecker factors. This representation is flexible as we can choose the number and the size of the Kronecker factors, which allows us to design models with the appropriate trade-off between computational efficiency and accuracy. The authors also introduced what they call a soft unitary constraint [START_REF] Cisse | Parseval networks: Improving robustness to adversarial examples[END_REF] which helps conditioning the matrix, especially in a recurrent network. A strict unitary constraint (i.e. projecting the matrix at each iteration so that it is orthogonal) would be computationally expensive and may deter the performances of the network. Kronecker products (or sum of Kronecker products) as also useful in the context of dictionary learning, [START_REF] Dantas | Learning dictionaries as a sum of kronecker products[END_REF][START_REF] Dantas | Learning tensor-structured dictionaries with application to hyperspectral image denoising[END_REF].

Kronecker

Architecture Design for Fast Inference

Techniques designed in previous Subsections 2.3.1 and 2.3.2 led the research community to focus on more efficient architectures that are trained from scratch. Such architectures are either designed by hand -following some ground principle -or using more advanced frameworks (architecture search or graph neural networks).

Efficient Architectures

One of the first deep neural networks, AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], contained 60 million parameters, including 10 million for the last three fully connected layers. The research community first searched for deeper and more parameter-efficient networks to ease the training on GPUs, using only convolutions with kernel sizes of 3 × 3 and 1 × 1 respectively49 (except for the first layer) and one single fully connected layer acting as a classifier. Such popular networks include the VGG [START_REF] Simonyan | Very deep convolutional networks for largescale image recognition[END_REF], Inception [START_REF] Szegedy | Going deeper with convolutions[END_REF], ResNets and ResNeXts (He et al., 2015a;[START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] models that are intensively used for various benchmarks.

Guided by the possible mobile applications, researchers then focused on networks that are able to run in real-time on a CPU at inference time. Such models, among which MobileNets [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF]Sandler et al., 2018a), ShuffleNets (Zhang et al., 2017b) or SqueezeNets [START_REF] Hu | Squeeze-and-excitation networks[END_REF] depth-wise50 and point-wise51 convolutional filters that require less parameters and floating-point operations than a traditional 3 × 3 convolution (see Table 2.3). These efficient architectures inspired the development of more efficient Transformers in NLP [START_REF] Vaswani | Attention is all you need[END_REF] such as MobileBERT [START_REF] Sun | MobileBERT: a compact task-agnostic bert for resource-limited devices[END_REF] as well as architecture adaptations by Wu et al. (2019b); Zhang et al. (2018a) and Sukhbaatar et al. (2019a).

Finally, the EfficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] and RegNet [START_REF] Radosavovic | Designing network design spaces[END_REF]) network families bridge the gap between GPU-efficient, yet heavy models and mobilecapable networks by proposing a range of models for various operating points. Their common denominator is to heavily rely almost exclusively on group convolutions and pointwise convolutions with a small -yet carefully tuned -number of channels that grows with depth. As depicted in Figure 2-13, the smallest networks have over 4 million parameters for a test top-1 error on ImageNet around 25%, to be compared with a ResNet-5052 that has 24 million parameters for a test top-1 error around 24%.

Feature Reuse and Wiring Patterns

Guided by the use of residual connections (He et al., 2015a), Huang et al. (2017a) designed a densely connected convolutional network (DenseNet) where each layer takes as input the output features of all the preceding layers. Such architectures are more parameter-efficient and yield good performance, in part since the skipconnections strongly encourage feature reuse through the upper layers, and allow us to easily train very deep networks (more than 100 stacked layers). For instance, a DenseNet-201 achieves 1.5% less top-1 error than a ResNet-50 for the same number of parameters. In the search for more efficiency, (Huang et al., 2018a) propose to replace the standard 3 × 3 convolutions used in the DenseNet by learnt 3 × 3 group convolutions in a network called the CondenseNet. Indeed, group convolutions are more parameter efficient and mobile-friendly than their traditional counterparts53 .

Feature reuse also powers the feature extractor of the Faster-R-CNN object detection model (Ren et al., 2015), which is a Feature Pyramid Network, allowing to extract information from the input image at different scales. We refer the reader to (Lin et al., 2017a) for further details.

Weight Sharing

Weight sharing is a simple technique assigning different parts of the network with the same value, for instance when two layers are assigned with the same weight matrix. During back-propagation, the gradients are summed54 so that the updated shared value remains the same. This method is used among others by [START_REF] Dehghani | Universal transformers[END_REF] or [START_REF] Lan | ALBERT: A lite BERT for self-supervised learning of language representations[END_REF] to produce parameter-efficient Transformers.

A natural extension of weight sharing is the Siamese Networks [START_REF] Bromley | Signature verification using a "siamese" time delay neural network[END_REF]. The idea is to share the weights or two entire blocks or networks, each of them being given a different type of input, in order to mutualize knowledge. Siamese Networks are used successfully to compare image patches [START_REF] Zagoruyko | Learning to compare image patches via convolutional neural networks[END_REF], for object tracking [START_REF] Bertinetto | Fully-convolutional siamese networks for object tracking[END_REF] or for joint learning of speaker and phonetic similarities [START_REF] Zeghidour | Joint learning of speaker and phonetic similarities with siamese networks[END_REF].

Neural Architecture Search

Feedforward Neural Networks are defined as directed acyclic graphs. In light of this generic definition, it appears that the traditional architectures represent a negligible proportion of all possible feedforward networks. Instead of relying on some based principles and designing the network essentially by hand, requiring some in-domain knowledge, [START_REF] Zoph | Neural architecture search with reinforcement learning[END_REF] propose to use reinforcement learning to generate the model descriptions of neural networks that maximize the reward -here the validation accuracy on a dataset of interest, followed by Liu et al. (2017a); [START_REF] Pham | Efficient neural architecture search via parameter sharing[END_REF]; Liu et al. (2019a) and [START_REF] Luo | Neural architecture optimization[END_REF]. Such approaches conducted [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF] to produce the NASNet models, [START_REF] Howard | Searching for mobilenetv3[END_REF] to produce MobileNet-v3, [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] to produce the aforementioned EfficientNet seed network or Chen et al. (2020a) to produce the AdaBERT models.

Following this recent progress, [START_REF] Xie | Exploring randomly wired neural networks for image recognition[END_REF] propose to loosen the constraints on the search space of wiring patterns by relying on various random graph models from graph theory. Going further, [START_REF] Radosavovic | On network design spaces for visual recognition[END_REF] and [START_REF] Radosavovic | Designing network design spaces[END_REF] propose to design design spaces combining the advantages of Neural Architecture Search (NAS) and manual design, leading to the RegNet network families. Finally, traditional NAS approaches require to train (at least partially) the generated network to get the reward at each time step of the reinforcement learning algorithm, which is extremely computationally intensive55 . To alleviate this issue, Wu et al. (2019a) propose to train a stochastic super-net that contains a set parallel blocks at each layer, from which one given is sampled during every forward pass.

On a side note, [START_REF] Press | Improving transformer models by reordering their sublayers[END_REF] design a "Sandwich Transformer" by testing some random reordering of the sublayers of a Transformer Network and demonstrate gains on machine translation compared to the standard interleaved transformer.

Distillation: Learning Form a Teacher

Distillation techniques boost the accuracy of a small student network by training it to mimic the output of a larger teacher network instead of training it from scratch [START_REF] Bucila | Model compression[END_REF][START_REF] Lei | Do deep nets really need to be deep?[END_REF][START_REF] Hinton | Distilling the knowledge in a neural network[END_REF].

KL Divergence

Assume the student is producing a vector of probabilities 𝑞 = (𝑞 1 , . . . , 𝑞 𝑛 ) and the teacher a vector of probabilities 𝑝 = (𝑝 1 , . . . , 𝑝 𝑛 ) 56 . Here, the student network will not be trained with a classical cross-entropy loss (defined in Equation (2.16)) but with a distillation loss measuring the distance between the two output distributions:

ℒ 𝐷 (𝑞, 𝑝) = 𝐷 𝐾𝐿 (𝑝||𝑞) = ∑ 𝑖 𝑝 𝑖 log ( 𝑝 𝑖 𝑞 𝑖 )
.

(2.15)

Intuitively, say the task is ImageNet classification for both the teacher and the student, and the input image is a picture of an apple and a banana, labeled as apple. When training the student network from scratch without distillation, the crossentropy loss will indicate that only a banana is present in the picture since the supervision comes from a one-hot encoding of the true class. However, when trained with distillation, the student network supervised with the vector of probabilities of the teacher network for this particular image that contains richer information. In particular, the class banana will be assigned with a small yet non-zero probability. Note that the cross-entropy loss is a particular case of the KL Divergence when the teacher probability vector corresponds to the one-hot encoding of the true class indexed by

𝑗: if 𝑝 𝑖 = 0 for 𝑖 ̸ = 𝑗 and 𝑝 𝑗 = 1, then 57 ℒ 𝐶 (𝑞, 𝑗) = 𝐷 𝐾𝐿 (𝑝||𝑞) = log ( 1 𝑞 𝑗 ) = -log(𝑞 𝑗 ). (2.16)
Sometimes, the student is trained with a combination ℒ of the traditional crossentropy loss ℒ 𝐶 , and of the distillation loss ℒ 𝐷 . Denoting 𝜆 > 0 a regularization parameter, ℒ(𝑞, 𝑝, 𝑗) = ℒ 𝐶 (𝑞, 𝑗) + 𝜆ℒ 𝐷 (𝑞, 𝑝). The teacher network is trained beforehand and stays fixed -it only provides the probability vectors -during the distillation.

Distillation at Every Layer

Following the distillation paradigm, [START_REF] Romero | Fitnets: Hints for thin deep nets[END_REF] propose to train a student network using not only the outputs but also the intermediate representations 𝑦 (ℓ) (𝑥) generated by the teacher network. Since the size of the intermediate representations -or features -of the student and the teacher or even the number of layers may not be equal, the authors introduce additional learnt parameters to map these rep-56 Such vectors are obtained by applying the softmax function to the output of the network. 57 With the convention that 0 log(0) = 0.

resentations and minimize the 𝐿 2 norm between them. Experiments conclude that guiding the training of the student network with intermediate "hints" provides faster and better convergence, as demonstrated subsequently for BERT model training by [START_REF] Sanh | DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[END_REF] or [START_REF] Jiao | Tinybert: Distilling BERT for natural language understanding[END_REF] and even for Generative Adversarial Networks (GAN) by [START_REF] Li | Gan compression: Efficient architectures for interactive conditional GANs[END_REF].

Semi-Supervised Training

When training a student network with the plain distillation loss ℒ 𝐷 , there is no need for labeled data, provided a trained teacher network is available. However, the teacher network might be trained with labeled data for particular tasks such as image classification. [START_REF] Yalniz | Billionscale semi-supervised learning for image classification[END_REF] exploit this idea and propose to train a largecapacity teacher network on a labeled dataset. The teacher is then used to label a large collection of unlabelled images (up to 1 billion) that are use to pre-train the student model, which is finally finetuned on the labelled dataset.

Scalar and Vector Quantization

Quantization generally refers to the idea of storing parameters or groups of parameters at a lower precision than the standard floating-point fp32 format58 -recall that fp32 numbers are stored over 32 bits or 4 bytes59 . The cornerstone of quantization is the 𝑘-means algorithm [START_REF] Steinhaus | Sur la division des corps materiels en parties[END_REF][START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF][START_REF] Lloyd | Least squares quantization in pcm[END_REF] used to cluster the parameters or groups of parameters by assigning every member of one cluster with the value of its centroid or codeword. This discretization helps reduce the network size and improve the inference time with specialized kernels and/or hardware as explained in Subsection 2.3.6, at the cost of a reduced accuracy. In what follows we work with a weight matrix 𝑊 = (𝑤 𝑘,𝑙 ) ∈ R 𝐶 in ×𝐶out and focus on the main training techniques for quantizing while maintaining a desired level of accuracy.

Floating-point Arithmetics

Before exposing the quantization methods per se, we briefly review the floating-point half-precision format or fp16 60 , which is similar to the fp32 format except that the numbers are stored over 16 bits. This format, that halves the size of the network, is becoming standard for storing and even for training Deep Learning models [START_REF] Micikevicius | Mixed precision training[END_REF]. Indeed, training with half-precision allows us to consider bigger networks or to train existing networks faster as detailed in Subsection 2.3.6. Since the range allowed by fp16 is smaller than fp32, the authors propose to stabilize the training (1) by dynamically scaling the loss so that the gradients of the activations fall within the acceptable range;

(2) by keeping a master fp32 copy61 of the network's parameters for the optimization step and (3) by leaving the BatchNorm layers in fp32 format. Many frameworks such as APEX62 now propose seamless integration of mixed precision training with no loss of accuracy compared to fp32.

Scalar Quantization

Fixed-point scalar quantization is a generic method that replaces the floating-point weights by 𝑁 bit fixed-point numbers [START_REF] Gupta | Deep learning with limited numerical precision[END_REF]. Fixed-point operations generally take less energy and area on chips than their floating-point counterparts (Han et al., 2016a;[START_REF] Lian | High-performance FPGA-based CNN accelerator with block-floating-point arithmetic[END_REF]. Setting 𝑁 = 8 defines the popular int8 quantization scheme while having 𝑁 = 1 encompasses the extreme case of binarization [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF]. More precisely, the parameters are rounded to one of 2 𝑁 possible codewords. Let us detail the case of uniform scalar quantization below.

Here, the codewords correspond to bins evenly spaced by a scale factor 𝑠 and shifted by a bias 𝑧. Each weight 𝑤 𝑘𝑙 is mapped to its nearest codeword 𝑐 by successively quantizing with 𝑧 ↦ → round(𝑤 𝑘𝑙 /𝑠 + 𝑧) and dequantizing with the inverse operation:

𝑐 = (round(𝑤 𝑘𝑙 /𝑠 + 𝑧) -𝑧) × 𝑠, (2.17)
where we compute the scale and bias from the trained fp32 weight statistics:

𝑠 = max 𝑊 -min 𝑊 2 𝑁 -1 and 𝑧 = round(min 𝑊/𝑠). (2.18)
This calibration step varies according to the quantization methods and will be discussed in the remainder of this Section. Note that the uniform rounding scheme in Equation (6.2) allows for fixed-point arithmetic with implementations in PyTorch and Tensorflow (see Subsection 2.3.6). The compression rate is 32/𝑁 and the activations 𝑦 (ℓ) (𝑥) are also generally rounded to 𝑁 -bit fixed-point numbers. In what follows, we first focus on the widely used case of int8 with 𝑁 = 8 and then detail compression schemes with lower precisions. Due to its wide range of applications, there is an extremely large body of literature on int8 quantization. For the sake of conciseness, we mention the salient trends and refer the reader to the two recent surveys [START_REF] Guo | A survey on methods and theories of quantized neural networks[END_REF][START_REF] Cheng | A survey of model compression and acceleration for deep neural networks[END_REF] for a comprehensive overview.

∘ Post-training Quantization. One straightforward baseline is that of post-training quantization, where the scale 𝑠 and the zero-point 𝑧 are calibrated63 as follows.

-The weight quantization parameters are calibrated offline as the weights are fixed, and per tensor as in Equation (2.18). There is also the possibility of quantizing per channel where we calibrate one scale and zero-point per channel, which diminishes the quantization error while presenting a small memory overhead64 , although it requires specialized kernels, see Subsection 2.3.6.

-The activation quantization parameters are calibrated using an exponential moving average (EMA) of the current quantization parameters when forwarding a few batches of data and maintaining the weights fixed. Since there may be some outliers in the activation values, considering the minimum and maximum values of the activations for calibration as detailed in Equation (2.18) may decrease the accuracy. Hence, there exist more sophisticated observers65 minimizing the quantization error measured as the 𝐿 2 norm between the quantized and non-quantized tensors.

∘ Quantization-Aware Training (QAT). This very strong baseline was designed by [START_REF] Krishnamoorthi | Quantizing deep convolutional networks for efficient inference: A whitepaper[END_REF]. The idea is to emulate int8 quantization of the weights and/or the activations when training the network to prepare the post-training quantization. The main under-the-hood take-away is to use the Straight-Through Estimator (STE) during the backward pass [START_REF] Bengio | Estimating or propagating gradients through stochastic neurons for conditional computation[END_REF][START_REF] Courbariaux | Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1[END_REF]. With STE, the gradients are obtained by replacing the non-quantized weights by their quantized counterparts during the backward pass (see 6.4.1).

∘ Learnable Scale and Zero-Point. Instead of calibrating the scale and zeropoint as explained in Equation (2.18), one possibility is to learn them through back-propagation. The main difficulty is to make the quantization operation dif-ferentiable with respect to 𝑤 and 𝑧, either by modeling the quantization noise66 [START_REF] Nagel | Data-free quantization through weight equalization and bias correction[END_REF] or by approximating the gradients [START_REF] Bhalgat | Lsq+: Improving low-bit quantization through learnable offsets and better initialization[END_REF].

∘ Non-uniform Schemes. The common idea of non-uniform quantization schemes [START_REF] Choi | Pact: Parameterized clipping activation for quantized neural networks[END_REF][START_REF] Li | Additive powers-of-two quantization: A nonuniform discretization for neural networks[END_REF] is to take advantage of the bell-shaped and long-tailed distribution of the weights and activations (Han et al., 2016b). Indeed, uniform quantization is optimal when the weights or activations follow a uniform distribution, which is not the case in practice. Here, by having adaptive bin sizes, the authors observe less degradation in the test accuracy. However, the methods are less prone to hardware acceleration, in contrast to uniform quantization.

∘ Mixed precision. There are extensive studies of scalar quantization to train networks with lower precision weights and activations, for instance [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF][START_REF] Courbariaux | Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1[END_REF][START_REF] Zhu | Trained ternary quantization[END_REF][START_REF] Zhou | Dorefanet: Training low bitwidth convolutional neural networks with low bitwidth gradients[END_REF][START_REF] Rastegari | Xnornet: Imagenet classification using binary convolutional neural networks[END_REF][START_REF] Mark | Training wide residual networks for deployment using a single bit for each weight[END_REF]. Note that precisions can be mixed (Wu et al., 2018) depending on the layer's sensitivity to quantization67 [START_REF] Gluska | Exploring neural networks quantization via layer-wise quantization analysis[END_REF], or by considering 4-bit weights and 8-bit activations for instance.

Vector Quantization

Vector Quantization (VQ) and Product Quantization (PQ) have been extensively studied in the context of nearest-neighbor search [START_REF] Jegou | Product quantization for nearest neighbor search[END_REF][START_REF] Ge | Optimized product quantization[END_REF][START_REF] Norouzi | Cartesian k-means[END_REF]. The idea is to decompose the original high-dimensional space into a product of subspaces that are quantized separately with a joint codebook.

To our knowledge, [START_REF] Gong | Compressing deep convolutional networks using vector quantization[END_REF] were the first to introduce these quantizers for neural network quantization, followed by [START_REF] Miguel | Model compression as constrained optimization, with application to neural nets[END_REF].

Compressing 𝑊 with Product Quantization (PQ) amounts to evenly split each column into 𝑚 subvectors of dimension 𝑑 and to learn a codebook on the resulting set of 𝑚𝐶 out subvectors68 . Then, each column of 𝑊 is quantized by mapping each of its subvectors to its nearest codeword in the codebook 𝒞 = {𝑐 1 , . . . , 𝑐 𝑘 } as illustrated in Figure 2-14. When 𝑚 is set to 1, PQ is equivalent to vector quantization (VQ) and when 𝑚 is equal to 𝐶 in , it is the scalar 𝑘-means algorithm. The main benefit of PQ is its expressivity: each column is mapped to a vector in the product 𝒞 = 𝒞 × • • • × 𝒞, thus PQ generates an implicit codebook of size 𝑘 𝑚 . As we will see in Chapters 5 and 6, employing this discretization off-the-shelf does not optimize the right objective function, and leads to a catastrophic drift of performance for deep networks. However, using a proper objective function and a careful finetuning of the codebook (Han et al., 2016b) yields state-of-the-art performance as explained in the aforementioned Chapters.

Hardware and Metrics

While the metrics of network size (measured in MB) and predictive performance (measured with some test accuracy) enjoy a wide consensus for measuring the compression efficiency, the inference time is subject to various metrics or proxys that depend on the target hardware. Here, we provide a few pointers to such metrics along with their (dis)advantages. We focus on inference (test time) for two reasons. First, deep learning models are currently mostly deployed to embedded devices or production servers for inference. Second, it is currently not possible to train a quantized neural network below half-precision: for instance, QAT (see Subsection 2.3.5) only emulates quantization 69 to prepare the post-training, true quantization. Therefore, training with low-precision ML 70 is a future challenge [START_REF] Kung | Term revealing: Furthering quantization at run time on quantized DNNs[END_REF][START_REF] Jeon | BiQGEMM: Matrix multiplication with lookup table for binarycoding-based quantized DNNs[END_REF].

Kernels

We first focus give a very high-level overview of the concept of kernels. Roughly speaking, a kernel is a hardware-dependent set of low-level instructions that perform standard linear algebra operations such as matrix multiplication or convolution in an 69 By successively quantizing and dequantizing weights and activations. 70 Low-precision Machine Learning. optimized way. The first baseline for accelerating neural network inference is to fuse kernels. For instance, at inference time, the sequence convolution-ReLU-BatchNorm 71 is fusable into a single kernel and saves in/out memory writing of the intermediary activations. This same logic is followed by TorchScript 72 , which is an intermediate, compiled representation of a PyTorch model that runs in high-performance environments such as C++. Finally, int8 libraries such as fbgemm 73 provide speedups compared to their float counterparts by performing the operations -for instance convolutions -in int8. Accumulation is performed in int32 and the weights are pre-packed so that their access in memory during the inner loop are adjacent.

Hardware

Fixed-point quantization methods such as int8 benefit from specialized hardware to also improve the runtime during inference [START_REF] Vanhoucke | Improving the speed of neural networks on cpus[END_REF]. Researchers then attempted to incorporate these hardware constraints when designing their compression algorithms, in particular in terms of latency: for instance, using an emulator of the target hardware (Wang et al., 2018a;[START_REF] Yang | Designing energy-efficient convolutional neural networks using energy-aware pruning[END_REF], or a proxy of the hardware inference time such as FLOPs (see next), or with the framework of reinforcement learning [START_REF] Howard | Searching for mobilenetv3[END_REF]. Since specialized hardware is generally more performant but allows for less flexibility in the choice of operations and is less widespread, there is a tradeoff depending on the target inference time, the target devices and the development time. Among the popular hardware present in embedded devices, we name a few, including: the CPU 74 , the DSP 75 , the GPU 76 , or dedicated accelerators for deep learning inference such as Apple's Neural Engine. Some modern chips include all these components, such as the SnapDragon 855 77 . Each computation unit comes with its own development toolkit and libraries 78 .

FLOPS and Other Metrics

Since measuring the hardware latency requires a consequent engineering effort as seen in the previous paragraph, some work consider proxy metrics such as the FLOPs or 71 https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html. 72 https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html. 73 https://github.com/pytorch/FBGEMM. 74 Central Processing Unit, the most widespread but the less specialized computation unit. 75 Digital Signal Processor, usually used to process real-world time series. 76 Present in most modern smartphones, offloads the CPU for image processing and ML inference. 77 Which also has a dedicated module for fast int8 inference. 78 For instance, models running on Apple's phone GPUs are converted into the CoreML format. FLoating-point Operations79 required for performing one forward pass. As others have noted [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF][START_REF] Figurnov | Spatially adaptive computation time for residual networks[END_REF][START_REF] Yang | Designing energy-efficient convolutional neural networks using energy-aware pruning[END_REF]Louizos et al., 2017a;He et al., 2018b), these metrics are far from perfect. Indeed, parameter count and FLOPs are a loose proxy for real-world latency, throughput, memory usage (RAM), and power consumption (battery drain). In other words, not all FLOPs are equal [START_REF] Dudziak | BRP-NAS: Prediction-based NAS using GCNs[END_REF]. Symmetrically, evaluating the latency of a deep learning model based only on the number of operations it is able to perform per second is considered harmful [START_REF] Sze | How to evaluate deep neural network processors: Tops/w (alone) considered harmful[END_REF]. As an illustration, we give an overview of the current empirical Pareto curve80 of computer vision models in terms of number of parameters, accuracy and latency in Figure 2-15 with no particular optimization (torchscript, int8) so only the relative value of the displayed numbers matters.

The Hardware Lottery

Finally, we close this Subsection with the concept of Hardware Lottery [START_REF] Hooker | The hardware lottery[END_REF]. In this essay, the author underlines the interdependency between existing hardware and fruitful research directions. Based on examples from computer science history, the author argues that hardware lotteries can delay research progress, especially given the advent of specialized hardware that increases the frictions when testing ideas that do not rely on this hardware.

Smaller Models, Bigger Biases? Hooker et al. (2020b) explored the impact of compression on image recognition models' ability to perform accurately across various human groups. In earlier work, Hooker et al. (2020a) showed that compressed image recognition models, although they maintained their accuracy overall, had trouble identifying classes that were rare in their training data. To learn whether that shortcoming translates into bias against underrepresented human types, the researchers trained models to recognize a particular class (people with blond hair), compressed them, and measured the differences in their accuracy across different types of people. This enabled them to point out some gaps in performance between compressed and uncompressed models with respect to underrepresented groups.

Conclusion

In this Chapter, we studied the over-parameterization of Neural Networks from the point of view of the deep learning practitioner, the engineer and the mathematician. While this parameter redundancy is at the core of Deep Learning and helps optimization, it presents training challenges that can be alleviated by introducing proper regularization and normalization techniques. Another way to take advantage of this redundancy is to group networks that behave identically and work in the quotient space. This idea relies on the concept of functional equivalence classes, that are in general surprisingly small and are generated only by permutations and rescalings of hidden neurons. Studying the redundancy of the network's parameters naturally leads to the question of the efficiency of such networks, hence to the question of their compression. We reviewed various compression methods, along with the success metrics, especially in terms of latency on a given target hardware. Although not completely orthogonal -for instance pruning can be viewed as architecture search and learned group convolutions as sparsity patterns -these methods can be combined and the question of selecting the right compression method (or combination thereof) for a given task and given target metrics is still an open problem.

Chapter 3

Group and Understand: Functional Equivalence Classes

In this Chapter, we aim at characterizing functional equivalence classes of neural networks. As illustrated with our contribution in Chapter 4, a better understanding of the structure of such classes would allow us to conveniently operate in the space of neural network parameterizations quotiented by the functional equivalence relation. We restrict ourselves to feedforward ReLU neural networks. We first list various definitions of the rescaling operation found in the literature and prove that they are all equivalent under the condition that each hidden neuron is connected to at least one input neuron and one output neuron. We then provide a stand-alone result for the onehidden-layer case where we prove that two functionally equivalent parameterizations are permutation-rescaling equivalent under some irreducibility conditions. Next, we investigate the case with many hidden layers by relying on the algebraic and geometric properties of the rescaling operation and by considering a local version of the rescaling equivalence. Finally, noticing that the case with deeper networks is not entirely solved, we list potential future directions and motivate them with the one-hidden-layer case.

Introduction

In this work, we study the over-parameterization of neural networks from the perspective of their functional equivalence classes. More precisely, we denote 𝑅 𝜃 the function implemented by a network parameterized with 𝜃 and call it the realization of the network. Then, two parameterizations 𝜃 and 𝜃 ′ are functionally equivalent if both realizations 𝑅 𝜃 and 𝑅 𝜃 ′ are equal. As explained in Section 2.2.5, members of the same functional equivalence class behave differently under the action of common methods used extensively in the Deep Learning field, for instance optimization or quantization algorithms. We argue that a better understanding of such classes may lead for instance to refined or novel optimization or quantization schemes, see Chapter 4.

We begin by recalling various definitions of rescaling equivalence for ReLU networks found in the literature as listed in Subsection 2.2.1, namely at the layer (Definition 2.2.2), neuron (Definition 2.2.3) and path level (Definition 2.2.4). Note that we introduce later in this Chapter our own definition of rescaling equivalence that we call trajectory-wise (Definition 3.3.4). Our first contribution is to reconcile these four definitions for admissible parameterizations, as proven in Appendix A.1.1. More formally, we denote by ∼ Layer (resp. ∼ Neuron , ∼ Path , ∼ Trajectory ) the equivalence relation induced by the layer (resp. neuron, path, trajectory) definition, and state the result in the following Proposition.

Proposition 3.1.1. Let 𝜃 be an admissible parameterization. Then, for every 𝜃 ′ , admissible or not, and every 𝑎, 𝑏 ∈ {Layer, Neuron, Path, Trajectory}, we have

𝜃 ∼ 𝑎 𝜃 ′ ⇐⇒ 𝜃 ∼ 𝑏 𝜃 ′ .
The admissibility assumption on 𝜃 is not used to prove that the layer-wise and neuron-wise definitions are equivalent, but is used to show all the other equivalences. We end this Section by recalling and proving a well-known result [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF], which states that functional equivalence classes contain orbits generated by the rescaling and permutation operations (proven in Appendix A.1.2). Proposition 3.1.2. Two parameterizations that are permutation-rescaling equivalent are functionally equivalent.

In the remainder of this Chapter, we investigate the reciprocal of Proposition 3.1.2: we aim at identifying irreducibility conditions under which two parameterizations that are functionally equivalent are also permutation-rescaling equivalent.

Reciprocal for One Hidden Layer

Irreducible Parameterizations

We first define irreducible parameterizations by introducing the concepts of dead, twin and co-dependant neurons for one hidden layer architectures. Definition 3.2.1 (Dead Neuron). Let 𝐺 be a one hidden layer architecture valued with 𝜃. Any hidden neuron 𝜈 ∈ 𝐻 is said to be dead if either 𝑤 •→𝜈 = 0 or 𝑤 𝜈→• = 0. Remark that 𝜃 is admissible if, and only if, there are no dead neurons.

Intuitively, a dead neuron does not participate in the realization 𝑅 𝜃 of the network. For 𝜃 admissible, we denote, for any 𝜈 ∈ 𝐻, Γ 𝜈 the affine hyperplane

Γ 𝜈 = {𝑥 ∈ R 𝑁 0 | ⟨𝑤 •→𝜈 , 𝑥⟩ + 𝑏 𝜈 = 0}.
Note that Γ 𝜈 is well-defined since 𝑤 •→𝜈 ̸ = 0 (𝜃 is admissible).

Definition 3.2.2 (Twin Neurons).

Let 𝐺 be a one hidden layer architecture valued with 𝜃 admissible. Any hidden neurons 𝜈 1 ̸ = 𝜈 2 ∈ 𝐻 are said to be twins if there exists 𝑑 ̸ = 0 such that 𝑤 •→𝜈 1 = 𝑑𝑤 •→𝜈 2 and 𝑏 𝜈 1 = 𝑑𝑏 𝜈 2 . Equivalently, the separating hyperplanes are equal:

Γ 𝜈 1 = Γ 𝜈 2 .
Intuitively, two twin neurons either (1) fire in the same half-space or (2) fire in complementary half-spaces. Note that we can properly define twin neurons in Definition 3.2.4 since there are no dead neurons, hence 𝜃 is admissible. Let 𝐺 be a one hidden layer architecture valued with 𝜃. We say that 𝜃 is irreducible if there are no dead, no twin and no co-dependent (hidden) neurons in the network.

Main Result

We now state our result, which is proven in Appendix A.1.3. In contrast to previous work summarized in Table 2.1 [START_REF] Kůrková | Functionally equivalent feedforward neural networks[END_REF][START_REF] Héctor | Uniqueness of the weights for minimal feedforward nets with a given input-output map[END_REF][START_REF] Albertini | Uniqueness of weights for neural networks[END_REF], we focus on the ReLU case and derive geometrical -hence easy to understand and to remember -irreducibility conditions. Proposition 3.2.1 provides sufficient conditions (irreducibility) such that functional equivalence implies permutation-rescaling equivalence. We now investigate the subtleties of non-irreducible parameters.

Subtleties of Non-Irreducible Parameters

We denote by the cases where it is possible to construct 𝜃 ′ from 𝜃 such that 𝑅 𝜃 = 𝑅 𝜃 ′ but 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent, and by the cases where it is not possible. We construct 𝜃 ′ from 𝜃 by setting 𝜃 ′ = 𝜃 except for some edges or neurons as specified in the cases below. For the sake of conciseness and clarity, we focus on examples to illustrate our point.

Dead Neurons.

Let us assume that 𝜃 has a dead hidden neuron 𝜈 2 as illustrated in Figure 3-2. Let 𝜀 > 0. Then, we set 𝑤 ′ 𝜇 2 →𝜈 2 = 𝑤 𝜇 2 →𝜈 2 + 𝜀. This modification does not change the realization 𝑅 𝜃 ′ = 𝑅 𝜃 , and is such that 𝜃 ′ ̸ ∼ 𝑃 𝑆 𝜃.

Twin Neurons. We specify the different types of twin neurons as follows:

• Siamese. The separating hyperplanes are oriented in the same direction. Let us assume that 𝜃 has two siamese neurons 𝜈 1 and 𝜈 2 as illustrated in Figure 3-3. Let 𝜀 > 0. Since the functions 𝑦 𝜈 1 and 𝑦 𝜈 2 are proportional, we set

𝑤 ′ 𝜈 1 →𝜂 = 𝑤 𝜈 1 →𝜂 + 𝜀 and 𝑤 ′ 𝜈 2 →𝜂 = 𝑤 𝜈 2 →𝜂 -𝜀 such that 𝑤 ′ 𝜈 1 →𝜂 + 𝑤 ′ 𝜈 1 →𝜂 = 𝑤 𝜈 1 →𝜂 + 𝑤 𝜈 1 →𝜂 .
Then, 𝑅 𝜃 ′ = 𝑅 𝜃 and 𝜃 ′ and 𝜃 are nor permutation-rescaling equivalent.

• Anti-Siamese. The separating hyperplanes are oriented in opposite directions.

-Invisible. -Visible The breakpoint is visible on the realization 𝑅 𝜃 .

∘ Cousin. There exists a third hidden neuron with a separating hyperplane that is parallel to Γ 𝜈 1 = Γ 𝜈 2 . Let us assume that 𝜃 has three cousin neurons 𝜈 1 , 𝜈 2 and 𝜈 3 as illustrated in Figure 3-5. Here, neurons 𝜈 1 and 𝜈 2 implement the same breakpoint 𝑥 𝜇 = 0 and 𝜈 3 the breakpoint 𝑥 𝜇 = 1. We define 𝜃 ′ such that 𝜈 1 and 𝜈 2 implement the same breakpoint 𝑥 𝜇 = 1 and 𝑛𝑢 3 the breakpoint 𝑥 𝜇 = 0. The realization is the same and the parameters are not permutation-rescaling equivalent.

∘ Non-Cousin. All the other hidden neurons have a separating hyperplane that is not parallel to Γ 𝜈 1 = Γ 𝜈 2 . In that case, it is not possible to construct 𝜃 ′ such that 𝑅 𝜃 = 𝑅 𝜃 ′ and such that 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent. We do not provide a proof for the general case case and state this example for illustrative purposes.

The Notion of Locality. As illustrated in Figures 3-2, 3-3 and 3-4 for dead, siamese and invisible neurons, it is possible to construct 𝜃 ′ arbitrarily close to 𝜃 such that 𝑅 𝜃 = 𝑅 ′ 𝜃 but 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent. This is not the case for cousin neurons. In the next sections, we will focus on local changes as they allow to manipulate the realization in a more algebraic way.

Algebraic and Geometric Tools

We now wish to state irreducibility conditions for deeper networks. However, the proof of Proposition 3.2.1 relies on geometrical arguments that are not readily adaptable for many hidden layers. Hence, we present in Section 3.3 some algebraic and geometrical to tackle the (local) case of deeper networks in Section 3.4.

An Algebraic Expression of 𝑅 𝜃 and its Consequences

We express the realization of the ReLU network 𝑅 𝜃 in a more algebraic way than the traditional definition relying on iterative composition of layer-wise functions of the type 𝑧 ↦ → 𝜎(𝑧𝑊 + 𝑏) (see Equation (2.9)). This allows us to formally state the well-known piecewise affine nature of 𝑥 ↦ → 𝑅 𝜃 (𝑥) and the lesser-known piecewise polynomial nature of 𝜃 ↦ → 𝑅 𝜃 (𝑥). A similar formula1 is stated without taking the biases into account in [START_REF] Meng | 𝒢-SGD: Optimizing relu neural networks in its positively scale-invariant space[END_REF] whereas [START_REF] Balduzzi | The shattered gradients problem: If resnets are the answer, then what is the question? ICML[END_REF] (Lemma A.2) perform analogous computations for gradient computations, still without biases.

As defined in the Notations, we denote 𝒫 the set of full paths connecting any input neuron to any output neuron and 𝒬 the set of partial paths from any hidden neuron to any output neuron. For simplicity, we will denote a full path 𝑝 = (𝜈 0 , . . . , 𝜈 𝐿 ) ∈ 𝒫 by 𝑝 = (𝑝 0 , . . . , 𝑝 𝐿 ). We denote 𝒬 ℓ = {(𝑝 ℓ , . . . , 𝑝 𝐿 ) | 𝑝 ∈ 𝒫} the set of partial paths starting from hidden layer 0 < ℓ < 𝐿. With a sight abuse of notation, we denote by 𝑞 0 the first neuron of the path 𝑞. Finally, recall that 𝑎 𝑞 (𝜃, 𝑥) is the activation status of the full or partial path 𝑞 ∈ 𝒫 ∪ 𝒬 1 ∪ • • • ∪ 𝒬 𝐿 given 𝜃 and 𝑥:

𝑎 𝑞 (𝜃, 𝑥) ∏ 𝜈∈𝑞∩𝐻 1(𝑦 𝜈 (𝜃, 𝑥) > 0), (3.2)
where we set ∏ ∅ = 1. We now state the formula in Proposition 3.3.1 and provide a complete derivation in Appendix A.2.1. Proposition 3.3.1. For any parameterization 𝜃 = (𝑤, 𝑏), any 𝑥 ∈ R 𝑁 0 and any output neuron 𝜈 ∈ 𝑁 𝐿 ,

𝑅 𝜃 (𝑥) 𝜈 = ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) + 𝐿 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ 𝑞 ℓ =𝜈 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥).
(3.3)

In Equation 3.3, the ReLU activation function is accounted for in the activation statuses 𝑎 𝑝 (𝜃, 𝑥) and 𝑎 𝑞 (𝜃, 𝑥), and the path-sum formulation allows us to better disentangle the dependency on the input 𝑥 and the dependency on the parameters 𝜃 in the terms 𝑥 𝑝 0 𝑣 𝑝 (𝜃). A necessary (but not sufficient) condition for the slope of the piecewise affine function 𝑥 ↦ → 𝑅 𝜃 (𝑥) to change is that at least one of the 𝑎 𝑝 (𝜃, 𝑥), 𝑝 ∈ 𝒫 ∪ 𝒬 toggles. We prevent such cases by restricting ourselves to the domain 𝒳 𝜃 defined below. Similarly, we will only consider the piecewise polynomial function 𝜃 ↦ → 𝑅 𝜃 (𝑥) for fixed inputs 𝑥 ∈ 𝒳 𝜃 . The proof can be found in Appendix A.2.1. Such a formula is useful for computing the gradient of 𝜃 ↦ → 𝑅 𝜃 (𝑥) as explained in Proposition A.2.1.

Proposition 3.3.2. Let 𝒰(𝑥, 𝜃) be the set of all neighborhoods 𝑈 ⊂ R 𝑁 0 × R 𝐸∪𝐻 of (𝑥, 𝜃). We define, for any parameterization 𝜃,

𝒳 𝜃 {𝑥 ∈ R 𝑁 0 | ∃𝑈 ∈ 𝒰(𝑥, 𝜃), ∀𝑝 ∈ 𝒫∪𝒬, ∀(𝑥 ′ , 𝜃 ′ ) ∈ 𝑈, 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃 ′ , 𝑥 ′ )}. (3.4)
Then, 𝒳 𝜃 is open. Moreover, let 𝜃 be a parameterization and 𝑥 ∈ 𝒳 𝜃 . Then, for each full or partial path 𝑝 ∈ 𝒫 ∪ 𝒬, the function

𝑥 ′ ↦ → 𝑎 𝑝 (𝜃, 𝑥 ′ ) (resp. 𝜃 ′ ↦ → 𝑎 𝑝 (𝜃 ′ , 𝑥)) is locally constant in the neighborhood of 𝑥 (resp. in the neighborhood of 𝜃).
Such a set if useful for instance for specifying where the gradient of 𝜃 ↦ → 𝑅 𝜃 (𝑥) is defined (see Proposition A.2.1). We next state a well-known result (Pascanu et al., 2013;[START_REF] Montúfar | On the number of linear regions of deep neural networks[END_REF] about the affine piece-wise nature of 𝑥 ↦ → 𝑅 𝜃 (𝑥) which we prove in Appendix A.2.1. We refer the reader to Subsection 2.2.2 for illustrations.

Parameterizations of the Form 𝜃

′ = 𝜃 ⊙ 𝑒 𝛾
In this Subsection, we investigate multiplicative changes of the form 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 in the parameter space, where ⊙ denotes pointwise multiplication, with the goal to study:

(1) The effect on the realization 𝑅 𝜃 ′ (in this Subsection);

(2) The conditions on 𝛾 such that the perturbed parameterization 𝜃 ′ is rescaling equivalent to 𝜃 (in the next Subsection 3.3.3).

We restrict ourselves to a network with scalar output and no output bias for the sake of simplicity. Before this, we first prove2 in Appendix A.2.2 that 𝛾 is uniquely defined for parameterizations 𝜃 ′ having the same sign as 𝜃 -we consider the extended sign as defined in the Notations which can take values in {-1, 0, 1} -provided that we impose3 the constraint 𝛾 ∈ Supp 𝜃 . We argue that considering such multiplicative changes instead of additive ones helps addressing both goals (1) and (2).

Proposition 3.3.5. Let 𝜃 be a parameterization. For all 𝜃 ′ in Sign 𝜃 , there exists a unique 𝛾 ∈ Supp 𝜃 such that 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 .

Next, we wish to simply express the realization 𝑅 𝜃 ′ when 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 is close enough to 𝜃. We begin by defining matrices indicating the presence or absence of a given edge 𝑒 in a full path 𝑝 or partial path 𝑞. Definition 3.3.1. We define P ∈ R 𝒫×𝐸 such that, for every edge 𝑒 ∈ 𝐸 and full path 𝑝 ∈ 𝒫, (P) 𝑝,𝑒 1(𝑒 ∈ 𝑝). Similarly, we define Q ∈ R 𝒬×𝐸 such that, for every edge 𝑒 ∈ 𝐸 and partial path 𝑞 ∈ 𝒬. (Q) 𝑞,𝑒 1(𝑒 ∈ 𝑞).

In order to consider only the non-zero terms in the two sums -full paths and partial paths -in Equation (3.3), we define P 𝜃 and Q 𝜃 as follows.

Definition 3.3.2. Given 𝜃 = (𝑤, 𝑏), we define

P 𝜃 Diag (1(𝑣 𝑝 (𝜃) ̸ = 0)) 𝑝∈𝒫 P. Q 𝜃 Diag (1(𝑏 𝑞 0 𝑣 𝑞 (𝜃) ̸ = 0)) 𝑞∈𝒬 Q. (3.5) (3.6)
Note that the term 𝑏 𝑞 0 only appears in Q 𝜃 .

We now wish to decouple the update applied to the weights -a simple multiplicative modification of the form exp(𝛼) -to the update applied to the biases which has the form exp(S𝛽) to make the bridge between the weights -the space 𝐸 of edges -and the biases -the space 𝐻 of hidden neurons. The proof is located in Appendix A.2.2. Reasons for treating weights and biases differently will become clearer in Subsection 3.3.3. Proposition 3.3.6. We define the linear map S : ker P → R 𝐻 such that

(S𝛽) 𝜈 - ∑ 𝑒∈𝑞 𝛽 𝑒 (3.7)
where 𝑞 is the partial path going from 𝜈 to the output neuron 𝜂. Then, 1. S is well-defined (i.e., independent of the choice of 𝑞);

2. S is an isomorphism and its inverse S -1 : R 𝐻 → ker P is such that for any 𝜆 ∈ R 𝐻 we have S -1 𝜆 = 𝛽 where for any edge

𝑒 = 𝜇 → 𝜈 ∈ 𝐸, 𝛽 𝑒 ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ -𝜆 𝜇 if 𝜇 ∈ 𝑁 𝐿-1 (and 𝜈 = 𝜂) 𝜆 𝜈 -𝜆 𝜇 if 𝜇 ∈ 𝑁 ℓ , 1 ≤ ℓ ≤ 𝐿 -2 𝜆 𝜈 if 𝜇 ∈ 𝑁 0 .
(3.8)

Now, consider 𝜃 = (𝑤, 𝑏) ∈ R 𝐸 × R 𝐻 a parameterization. We now refine our definition of 𝛾 given in Proposition 3.3.5 by expressing it either as 𝛾 ∈ Supp 𝜃 or by 𝛾 = (𝛼, S𝛽) where S is defined in Proposition3.3.6, with (𝛼, 𝛽) ∈ 𝑉 𝜃 , where

𝑉 𝜃 Supp 𝑤 ×S -1 (Supp 𝑏 ) ⊂ R 𝐸 × ker P ⊂ R 𝐸 × R 𝐸 .
(3.9)

Finally, we establish an explicit and local formula to describe trajectories near 𝜃 in the direction 𝛾, which we prove in Appendix A.2.2. We use the set 𝒳 𝜃 defined in (3.4) and start from the path-sums formula (3.3).

Proposition 3.3.7. Consider 𝜃 a parametrization and 𝐵 𝜃 {𝛾 ∈ Supp 𝜃 | ‖𝛾‖ 2 ≤ 1}. Then, for each 𝑥 ∈ 𝒳 𝜃 , there exists 𝜀 > 0 such that, for every (𝛾, 𝑡) ∈ 𝐵 𝜃 ×] -𝜀, 𝜀[, the following result holds:

𝑅 𝜃 (𝜃 ⊙ 𝑒 𝛾𝑡 , 𝑥) = ∑ 𝑝∈𝒫 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑒 𝑡P(𝛼)𝑝 𝑎 𝑝 (𝜃, 𝑥) + ∑ 𝑞∈𝒬 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑒 𝑡Q(𝛼-𝛽)𝑞 𝑎 𝑞 (𝜃, 𝑥). (3.10)
where we recall that 𝛾 = (𝛼, S𝛽).

Recalling Definition 3.3.2, we note that the terms P𝛼 and Q(𝛼-𝛽) can be replaced in Equation (3.10) by P 𝜃 (𝛼) and Q 𝜃 (𝛼 -𝛽) respectively without changing the result since 𝑣 𝑝 (𝜃) = 0 implies that 𝑃 𝜃 (𝛼) 𝑝 = 0 for any full path 𝑝 ∈ 𝒫 and similarly for any partial path 𝑞 ∈ 𝒬.

Algebraic Characterization of Rescaling Equivalence

Given an admissible parameterization 𝜃 = (𝑤, 𝑏), we characterize the set of multiplicative changes 𝑒 𝛾 with respect to 𝜃 that preserve the rescaling equivalence when applied to 𝜃, leading to a more geometrical definition of rescaling equivalence in Definition 3.3.4, which we illustrate in Subsection A.2.4. Definition 3.3.3. Let 𝜃 be a parameterization. The vector 𝛾 = (𝛼, S𝛽) ∈ Supp 𝜃 is said to be 𝜃-compatible or compatible when the dependency on 𝜃 is obvious if 𝛼 ∈ ker (P 𝜃 ) and 𝛼 -𝛽 ∈ ker (Q 𝜃 ) .

(3.11)

We denote by Comp 𝜃 the set of 𝜃-compatible vectors 𝛾. With a slight abuse of notation, we might also say that (𝛼, 𝛽) ∈ 𝑉 𝜃 is 𝜃-compatible.

Remark 3.3.1. We will constantly distinguish between the case without biases and the case with biases for the rest of this section.

• If 𝑏 = 0, the condition 𝛼-𝛽 ∈ ker (Q 𝜃 ) is always satisfied using Definition 3.3.2.

• If 𝑏 ̸ = 0, the condition 𝛼 -𝛽 ∈ ker (Q 𝜃 ) ensures that the rescaling coefficients corresponding to non-zero biases 𝑏 𝜈 are the same as the rescaling coefficients obtained for the weights for neuron 𝜈.

We now define rescaling equivalence using the previous definition 3.3.3. We prove that this definition 3.3.4 is equivalent to the three definitions of layer-, neuron-and path-wise rescaling equivalence in Proposition 3.1.1 under the condition that the considered parameterizations are admissible. We finally define the set Sincomp 𝜃 of strongly incompatible vectors 𝛾 for reasons that will become clearer in Section 3.4. Definition 3.3.5. The vector 𝛾 = (𝛼, S𝛽) ∈ Supp 𝜃 is said to be strongly 𝜃-incompatible or simply strongly incompatible if 𝛼 ∈ ker (P 𝜃 ) ⊥ and ‖𝛾‖ 2 = 1.

(3.12)

We denote by Sincomp 𝜃 the set of strongly 𝜃-incompatible vectors 𝛾.

We prove the following properties on Sincomp 𝜃 in Appendix A.2.3.

Proposition 3.3.8. We denote by Comp 𝜃 the complementary set of Comp 𝜃 . Then, the two following properties hold for Sincomp 𝜃 .

(1) Sincomp 𝜃 ⊂ Comp 𝜃 for any admissible parameterization 𝜃.

(2) Sincomp 𝜃 is compact for any parameterization 𝜃.

We provide an illustration of the tools designed in this Section in Appendix A.2.4 for particular networks (Stem and Sawtooth) to showcase their usefulness.

Locally Identifiable Parameterizations

We now investigate a possible reciprocal of Proposition 3.1.2 using the tools designed in Section 3.3. We study parameterizations for which the property that functional equivalence implies rescaling equivalence holds locally. We first give definitions of locally identifiable and non-identifiable parameterizations in Subsection 3.4.1 and derive a necessary condition as well as a sufficient condition for local identifiability in Subsections 3.4.2 and 3.4.3.

Definition of Locally Identifiable Parameterizations

We first give an intuitive definition of locally identifiable parameters for which the property that functional equivalence implies rescaling equivalence holds locally. Definition 3.4.1. Consider 𝜃 an admissible parameterization.

• 𝜃 is locally identifiable if there exists a neighborhood Ω of 𝜃 such that for all

𝜃 ′ ∈ Ω, 𝑅 𝜃 ′ = 𝑅 𝜃 implies 𝜃 ′ ∼ 𝑆 𝜃.
• 𝜃 is locally non-identifiable if it is not locally identifiable, i.e. for all neighborhood Ω of 𝜃, there exists 𝜃 ′ ∈ Ω such that 𝑅 𝜃 ′ = 𝑅 𝜃 and 𝜃 ′ ̸ ∼ 𝑆 𝜃.

Let us now study the notions of locally non-identifiable and restricted locally non-identifiable parameterizations in order to leverage our geometrical definition of rescaling equivalence in Definition 3.3.3 that uses multiplicative changes defined in Proposition 3.3.5. Indeed, multiplicative changes only operate at constant sign. Note that a locally non-identifiable parameterization is not necessarily restricted locally non-identifiable: consider an architecture with two input neurons 𝜇 1 , 𝜇 2 , two hidden neurons 𝜈 1 , 𝜈 2 , two output neurons 𝜂 1 , 𝜂 2 and set:

𝑤 𝜇 1 →𝜈 1 = 1, 𝑤 𝜇 1 →𝜈 2 = 1, 𝑤 𝜇 2 →𝜈 1 = -1, 𝑤 𝜇 2 →𝜈 2 = -1 and 𝑤 𝜈 1 →𝜂 1 = 1, 𝑤 𝜈 1 →𝜂 2 = 0 and 𝑤 𝜈 2 →𝜂 1 = 0, 𝑤 𝜈 2 →𝜂 2 = 1.
Here, 𝜃 is locally non-identifiable. Indeed, neurons 𝜈 1 and 𝜈 2 are twins (see the example of Figure 3-3). However, 𝜃 is not restricted locally non-identifiable since we cannot get 𝜃 ′ such that: 𝜃 ′ has the same support as 𝜃, 𝜃 ′ ̸ ∼ 𝑆 𝜃 and 𝑅 𝜃 = 𝑅 𝜃 ′ .

We now provide an equivalent definition of restricted locally non-identifiable parameterizations in terms of strongly incompatible multiplicative changes (see Definition 3.3.5). The proof is given in Appendix A.3.1. Proposition 3.4.2. Consider 𝜃 = (𝑤, 𝑏) an admissible parameterization. The following properties are equivalent:

1. 𝜃 is restricted locally non-identifiable.

2. for all 𝜀 > 0, there exists 𝜏 ∈ ]0, 𝜀[ and 𝛾 ∈ Sincomp 𝜃 such that 𝑅 𝜃⊙𝑒 𝛾𝜏 = 𝑅 𝜃 .

Sufficient Condition for Restricted Local Identifiability

We now provide a necessary condition for an admissible parameterization 𝜃 to be a restricted locally non-identifiable parameterization, i.e. a sufficient condition for 𝜃 to be restricted locally identifiable. The proof can be found in Appendix A.3.2. Proposition 3.4.3. Let 𝜃 = (𝑤, 𝑏) be an admissible, restricted locally non-identifiable parameterization. Then, there exists 𝛾 ∈ Sincomp 𝜃 such that, for all 𝑥 ∈ 𝒳 𝜃 , ⟨∇ 𝜃 𝑅 𝜃 (𝑥), 𝜃 ⊙ 𝛾⟩ = 0.

(3.13)

Next, let us contextualize Proposition 3.4.3 for the one-hidden-layer case. Recall that our result in Proposition 3.2.1) states that, for irreducible parameterizations, functional equivalence implies permutation-rescaling equivalence. In particular, irreducible parameterizations are locally identifiable. In particular, as proven in Appendix A.3.2, restricted locally non-identifiable parameterizations are non irreducible. Proposition 3.4.4. Let 𝐺 be a one hidden layer architecture with scalar output valued with an admissible parameterization 𝜃 = (𝑤, 𝑏) such that 𝑏 = 0 (no biases). We assume that 𝜃 is restricted locally non-identifiable. Then there exists at least two twin hidden neurons as defined in Definition 3.2.2.

Sufficient Condition for Local Identifiability

We now adopt a pure geometrical point of view to provide another sufficient condition for Locally Identifiable (but non necessarily restricted) parameterizations in Proposition 3.4.7. Here again, we focus on the scalar output case.

Definition 3.4.3. Let 𝜃 be a network parameterization and 𝑥 ∈ 𝒳 𝜃 . We define 𝑢(𝜃) ∈ R 𝒫∪𝒬 such that, for all 𝑞 ∈ 𝒫 ∪ 𝒬,

𝑢 𝑞 (𝜃) = ⎧ ⎪ ⎨ ⎪ ⎩ 𝑣 𝑞 (𝜃) if 𝑞 ∈ 𝒫 𝑏 𝑞 0 𝑣 𝑞 (𝜃) if 𝑞 ∈ 𝒬.
(3.14)

For each 𝑥 ∈ 𝒳 𝜃 , we further define 𝑐(𝜃, 𝑥) ∈ R 𝒫∪𝒬 such that, for all 𝑞 ∈ 𝒫 ∪ 𝒬,

𝑐 𝑞 (𝜃, 𝑥) = ⎧ ⎪ ⎨ ⎪ ⎩ 𝑥 𝑞 0 𝑎 𝑞 (𝜃, 𝑥) if 𝑞 ∈ 𝒫 𝑎 𝑞 (𝜃, 𝑥) if 𝑞 ∈ 𝒬.
(3.15)

With these notations and recalling that we focus on the scalar output case, we have, for all 𝑥 ∈ R, 𝑅 𝜃 (𝑥) = ⟨𝑢(𝜃), 𝑐(𝜃, 𝑥)⟩.

We now define useful sets to only manipulate finitely many input points, as proven in Appendix A.3.3. Proposition 3.4.5. Let 𝜃 be a parameterization. There exists a neighborhood Ω 𝜃 of 𝜃 and a finite set of points 𝒵 𝜃 ⊂ 𝒳 𝜃 that satisfy the following properties:

• The following equality holds

Span 𝑧∈𝒵 𝜃 (𝑐(𝜃, 𝑧)) = Span 𝑥∈𝒳 𝜃 (𝑐(𝜃, 𝑥)).
(3.16)

• For all 𝜃 ′ ∈ Ω 𝜃 and for all 𝑧 ∈ 𝒵 𝜃 , 𝑐(𝜃 ′ , 𝑧) = 𝑐(𝜃, 𝑧).

We now express functional equivalence for two close parameterizations in pure geometrical terms in the space of full and partial paths, as proven in Appendix A.3.3. Proposition 3.4.6. Let 𝜃 be a parameterization, and Ω 𝜃 , 𝒵 𝜃 given as in Proposition 3.4.5. Denote by 𝒞 𝜃 the vector space

𝒞 𝜃 = Span 𝑥∈𝒳 𝜃 (𝑐(𝜃, 𝑥)).
(3.17)

Then, for any

𝜃 ′ ∈ Ω 𝜃 , if 𝑅 𝜃 = 𝑅 𝜃 ′ , then 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 .
Recall that in Definition 2.2.4, we expressed rescaling equivalence between 𝜃 and 𝜃 ′ as 𝑢(𝜃) = 𝑢(𝜃 ′ ) (using notations from Definition 3.4.3). We are now ready to state a sufficient condition ensuring that 𝜃 is locally identifiable relying on geometrical considerations in the space of the full and partial paths. The corresponding Proposition is proven in Appendix A.3.3.

Proposition 3.4.7. Let 𝜃 be a parameterization. We assume that there exists a neighborhood Ω of 𝜃 such that, for all 𝜃 ′ ∈ Ω, 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 implies that 𝑢(𝜃) = 𝑢(𝜃 ′ ). Then, 𝜃 is locally identifiable.

A simple corollary of Proposition 3.4.7 is given below and proven in Appendix A.3.3. Proposition 3.4.8. Let 𝜃 be an admissible parameterization such that 𝒞 ⊥ 𝜃 = {0}. Then, 𝜃 is locally identifiable.

Next, we leverage our standalone proof for the one-hidden-layer case showing that, for irreducible parameterizations, functional equivalence implies permutationrescaling equivalence (Proposition 3.2.1) to check for consistency. We apply Proposition 3.4.7 to the one hidden layer case (see Proposition 3.2.1) to verify that any irreducible parameterization is indeed locally identifiable. We first prove the following Proposition in Appendix A.3.3. Proposition 3.4.9. Let 𝐺 be a one hidden layer architecture with scalar output valued with an admissible parameterization 𝜃. We further assume that there are no twin neurons, i.e. that 𝜃 is irreducible as defined in 3.2.4. Then, 𝒞 ⊥ 𝜃 = {0}.

Using Proposition 3.4.8, we deduce that the parameterization 𝜃 given in Proposition 3.4.9 is locally identifiable. Moreover, taking a closer look at the proofs, we conjecture that if would be possible to obtain a characterization of locally identifiable parameterizations with the condition 𝒞 ⊥ 𝜃 = {0} for one-hidden-layer networks.

Current Limitations and Discussions

We list below the limitations of our current approach and formulate various conjectures that deserve further interest. First and foremost, Propositions 3.4.7 and 3.4.3 only encompass the case of local identifiability. Hence, even if we were to properly characterize locally identifiable parameterizations, we still wouldn't take the permutations into account or non-local degenerate cases as listed in Subsection 3.2.3. We argue that studying global identifiability is out of the scope of this work.

Next, we list a few directions that would improve our local understanding of local identifiability, namely we wish to (1) work in the path space and only manipulate Im(𝑢) in Proposition 3.4.7 and (2) sample a finite set of input points 𝒵 𝜃 such that, if 𝜃 and 𝜃 ′ are close enough and coincide on 𝒵 𝜃 , they coincide everywhere, i.e. 𝑅 𝜃 = 𝑅 𝜃 ′ . Using this fact, we would have a necessary and sufficient condition for local identifiability in Proposition 3.4.7. Before jumping into the details, we mention a similar line of work in this direction [START_REF] Malgouyres | Multilinear compressive sensing and an application to convolutional linear networks[END_REF][START_REF] Malgouyres | On the stable recovery of deep structured linear networks under sparsity constraints[END_REF], where the authors consider structured linear networks, without biases and without activation functions as well as per-layer (instead of neuron-wise) rescalings. See Section 2.2.4 in the Related Work for details. In particular [START_REF] Malgouyres | Multilinear compressive sensing and an application to convolutional linear networks[END_REF] also manipulate the quantities close to 𝑢(𝜃) and Im(𝑢).

First, assuming that a 𝜃 satisfies a certain condition (*), we adopt a more geometrical view about the manifold Im(𝑢) -𝑢(𝜃) (in the space of paths R 𝒫 ) relatively to 𝒞 𝜃 in the neighborhood of 𝑢(𝜃). The proof is given in Appendix A.3.4. Note that the knowledge of 𝑢(𝜃) allows to recover 𝜃 up to rescalings, see Definition 2.2.4. Proposition 3.4.10. Assume that 𝜃 satisfies the following property

∀𝜀 > 0, ∃𝜂 > 0, ∀𝜃 ′ , ‖𝑢(𝜃 ′ ) -𝑢(𝜃)‖ < 𝜂 =⇒ ∃𝜃 ′′ , 𝑢(𝜃 ′′ ) = 𝑢(𝜃 ′ ), ‖𝜃 ′′ -𝜃‖ < 𝜀 (*)
Then, the two following properties are equivalent:

(i) There exists 𝜀 > 0 such that, for all 𝜃 ′ ∈ 𝐵(𝜃, 𝜀) such that 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 , we have 𝑢(𝜃 ′ ) = 𝑢(𝜃).

(ii) There exists 𝜂 > 0 such that (Im(𝑢)

-𝑢(𝜃)) ∩ 𝐵 ∞ (0, 𝜂) ∩ 𝒞 ⊥ 𝜃 = {0}.
Next, we obtain necessary and sufficient condition on local identifiability, provided that our sampling conjecture (item (1) in the following Proposition) holds. The Proposition is proven in Appendix A.3.4. Proposition 3.4.11. Let assume that 𝜃 satisfies (*). Let us assume further that there exists 𝜀, 𝑟 > 0 and a finite set 𝒵 𝜃 ⊂ 𝒳 𝜃 such that Z𝜃 ∪ 𝑧∈𝒵 𝜃 𝐵(𝑧, 𝑟) ⊂ 𝒳 𝜃 and such that, for all 𝜃 ′ ∈ 𝐵(𝜃, 𝜀),

(1) If for all 𝑧 ∈ Z𝜃 , 𝑅 𝜃 (𝑧) = 𝑅 𝜃 ′ (𝑧), then 𝑅 𝜃 = 𝑅 𝜃 ′ .
(2) For all 𝑧 ∈ Z𝜃 , 𝑐(𝜃 ′ , 𝑧) = 𝑐(𝜃, 𝑧)

Then the two following properties are equivalent:

(i) 𝜃 is locally identifiable. (ii) There exists 𝜂 > 0 such that (Im(𝑢) -𝑢(𝜃)) ∩ 𝐵 ∞ (0, 𝜂) ∩ 𝒞 ⊥ 𝜃 = {0}.
We believe that Property (*) holds for any parameterization 𝜃. However, it is still a conjecture at this time. We provide below concrete examples of parameterizations 𝜃 that satisfy (*), as proven in Appendix A.3.4. Proposition 3.4.12. If all coefficients of 𝜃 = (𝑤, 𝑏) are strictly positive, then (*) defined in Proposition 3.4.10 holds.

We conjecture that assumption (1) of Proposition 3.4.11 holds for any parameterization 𝜃. On the other hand, assumption (2) is already proven in Proposition 3.4.5.

Conclusion

In this Chapter, we studied functional equivalence classes of ReLU neural networks. We first reconciled the various definitions of rescaling equivalence found in the literature under the condition that the underlying parameterizations are admissible. We then provided sufficient conditions -namely no dead or twin hidden neurons -in the one-hidden-layer case such that functional equivalence implies permutation-rescaling equivalence. Next, to investigate the case of deeper networks, we designed algebraic and geometrical tools to characterize the rescaling equivalence with multiplicative changes in the parameter space. Finally, we provide a sufficient condition for restricted local identifiability and another sufficient condition for local identifiability. Acknowledging that the case with deeper network is not entirely solved, we motivate and contextualize these conditions with particular networks and list interesting directions for further research.

Chapter 4

Learning to Balance the Energy with Equi-normalization

In this Chapter, we propose an application of the theoretical considerations on functional equivalence classes in Chapter 3. We restrict ourselves to describing such classes with the sole rescalings (see for instance Definition 2.2.2) for practicality. As proven in Proposition A.1.1 and illustrated in Figure 4-1, rescalings preserve the function implemented by the network and allow us to manipulate networks by considering their equivalence classes or orbits. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for selecting, within one equivalence class, the representant that minimizes the 𝐿 2 norm of its weights. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy and amounts to perform SGD in the set of parameters quotiented by the rescaling equivalence relation. For small batches, our approach offers an alternative to batch and group normalization on CIFAR-10 and ImageNet with a ResNet-18.

Introduction

Deep Neural Networks (DNNs) have achieved outstanding performance across a wide range of empirical tasks such as image classification [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], image segmentation (He et al., 2017a), speech recognition (Hinton et al., 2012a), natural language processing [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF] or playing the game of Go [START_REF] Silver | Mastering the game of go without human knowledge[END_REF]. These successes have been driven by the availability of large labeled datasets such as ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF], increasing computational power and the use of deeper models (He et al., 2015a).

Figure 4-1: Matrices 𝑊 (ℓ) and 𝑊 (ℓ+1) are updated by multiplying the columns of the first matrix with rescaling coefficients. The rows of the second matrix are inversely rescaled to ensure that the product of the two matrices is unchanged. The rescaling coefficients are strictly positive to ensure functional equivalence when the matrices are interleaved with ReLUs. This rescaling is applied iteratively to each pair of adjacent matrices. In this Chapter, we address the more complex cases of biases, convolutions, max-pooling or skip-connections to be able to balance modern CNN architectures. This Figure is the layer analogous of neuron-wise rescalings depicted in Figure 234.

Although the expressivity of the function computed by a neural network grows exponentially with depth (Pascanu et al., 2013;[START_REF] Telgarsky | Benefits of depth in neural networks[END_REF], in practice deep networks are vulnerable to both over-and underfitting [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]He et al., 2015a). Widely used techniques to prevent DNNs from overfitting include regularization methods such as weight decay [START_REF] Krogh | A simple weight decay can improve generalization[END_REF], Dropout (Hinton et al., 2012b) and various data augmentation schemes [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for largescale image recognition[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF]. Underfitting can occur if the network gets stuck in a local minima, which can be avoided by using stochastic gradient descent algorithms [START_REF] Bottou | Large-scale machine learning with stochastic gradient descent[END_REF][START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF][START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF][START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], sometimes along with carefully tuned learning rate schedules (He et al., 2015a;[START_REF] Goyal | Accurate, large minibatch sgd: Training imagenet in 1 hour[END_REF].

Training deep networks is particularly challenging due to the vanishing/exploding gradient problem. It has been first studied for Recurrent Neural networks (RNNs) [START_REF] Hochreiter | Gradient flow in recurrent nets: the difficulty of learning long-term dependencies[END_REF] as well as standard feedforward networks (He et al., 2015c;[START_REF] Mishkin | All you need is a good init[END_REF]. After a few iterations, the gradients computed during backpropagation become either too small or too large, preventing the optimization scheme from converging. This is alleviated by using non-saturating activation functions such as rectified linear units (ReLUs) [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] or better initialization schemes preserving the variance of the input across layers [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF][START_REF] Mishkin | All you need is a good init[END_REF]. Failure modes that prevent the training from converging have been theoretically studied by [START_REF] Hanin | How to start training: The effect of initialization and architecture[END_REF].

Two techniques in particular were key for vision models to achieve "super-human" accuracy. Batch Normalization (BN) was developed to train Inception networks [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. It introduces intermediate layers that normalize the features by the mean and variance computed within the current batch. BN is effective in reducing training time, provides better generalization capabilities after training and diminishes the need for a careful initialization. Network architectures such as ResNet (He et al., 2015a) and DenseNet (Huang et al., 2017a) use skip connections along with BN to improve the information flow during both the forward and backward passes.

However, BN has some limitations. In particular, BN only works well with sufficiently large batch sizes [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF][START_REF] Wu | Group normalization[END_REF]. For sizes below 16 or 32, the batch statistics have a high variance and the test error increases significantly. This prevents the investigation of higher-capacity models because large, memory-consuming batches are needed in order for BN to work in its optimal range. In many use cases, including video recognition [START_REF] Carreira | Quo vadis, action recognition? A new model and the kinetics dataset[END_REF] and image segmentation (He et al., 2017a), the batch size restriction is even more challenging because the size of the models allows for only a few samples per batch. Another restriction of BN is that it is computationally intensive, typically consuming 20% to 30% of the training time. Variants such as Group Normalization (GN) [START_REF] Wu | Group normalization[END_REF] cover some of the failure modes of BN.

In this Chapter, we introduce a novel algorithm to improve both the training speed and generalization accuracy of networks by using their over-parameterization to regularize them. In particular, we focus on neural networks that are positive-rescaling equivalent [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF], i.e. whose weights are identical up to positive scalings and matching inverse scalings. The main principle of our method, referred to as Equi-normalization (ENorm), is illustrated in Figure 4-1 for the fully-connected case. We rescale two consecutive matrices in order to minimize the joint 𝑝-norm of these two matrices under the constraint of preserving the function implemented by the network. We conjecture that this particular choice of rescaling coefficients ensures a smooth propagation of the gradients during training.

Related Work

This section reviews methods improving neural network training and compares them with ENorm. Since there is a large body of literature in this research area, we focus on the works closest to the proposed approach and refer the reader to Section 2.1.

From early works, researchers have noticed the importance of normalizing the input of a learning system, and by extension the input of any layer in a DNN (LeCun et al., 1998a). Such normalization is applied either to the weights or to the activations. On the other hand, several strategies aim at better controlling the geometry of the weight space with respect to the loss function. Note that these research directions are not orthogonal. For example, explicitly normalizing the activations using BN has smoothing effects on the optimization landscape (Santurkar et al., 2018b).

Normalizing Activations. Batch Normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] normalizes the activations by using statistics computed along the batch dimension. As stated in the introduction, the dependency on the batch size leads BN to underperform when small batches are used. Batch Renormalization (BR) [START_REF] Ioffe | Batch renormalization: Towards reducing minibatch dependence in batch-normalized models[END_REF]) is a follow-up that reduces the sensitivity to the batch size, yet does not completely alleviate the negative effect of small batches. Several batch-independent methods operate on other dimensions, such as Layer Normalization (channel dimension) [START_REF] Lei Ba | Layer normalization[END_REF] and Instance-Normalization (sample dimension) [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF]. Parametric data-independent estimation of the mean and variance in every layer is investigated by [START_REF] Arpit | Normalization propagation: A parametric technique for removing internal covariate shift in deep networks[END_REF]. However, these methods are inferior to BN in standard classification tasks. More recently, Group Normalization (GN) [START_REF] Wu | Group normalization[END_REF], which divides the channels into groups and normalizes independently each group, was shown to effectively replace BN for small batch sizes on vision tasks.

Normalizing Weights. Early weight normalization techniques only served to initialize the weights before training [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]He et al., 2015c). These methods aim at keeping the variance of the output activations close to one along the whole network, but the assumptions made to derive these initialization schemes may not hold as training evolves. More recently, [START_REF] Salimans | Weight normalization: A simple reparameterization to accelerate training of deep neural networks[END_REF] propose a polar-like re-parametrization of the weights to disentangle the direction from the norm of the weight vectors. Note that Weight Norm (WN) does require mean-only BN to get competitive results as well as a greedy layer-wise initialization.

Optimization Landscape. Generally, in the parameter space, the loss function moves quickly along some directions and slowly along others. To account for this anisotropic relation between the parameters of the model and the loss function, natural gradient methods have been introduced [START_REF] Amari | Natural gradient works efficiently in learning[END_REF]. They require storing and inverting the 𝑁 × 𝑁 curvature matrix, where 𝑁 is the number of network parameters.

Several works approximate the inverse of the curvature matrix to circumvent this problem (Pascanu and Bengio, 2013;[START_REF] Marceau-Caron | Practical riemannian neural networks[END_REF][START_REF] Martens | Optimizing neural networks with kroneckerfactored approximate curvature[END_REF]. Another method called Diagonal Rescaling [START_REF] Lafond | Diagonal rescaling for neural networks[END_REF] proposes to tune a particular re-parametrization of the weights with a block-diagonal approximation of the inverse curvature matrix. [START_REF] Qi | Deep isometric learning for visual recognition[END_REF] propose to enforce near isometric weights for both initialization and training. Finally, [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF] propose a rescaling invariant path-wise regularizer and use it to derive Path-SGD, an approximate steepest descent with respect to the path-wise regularizer.

Positioning. Unlike BN, Equi-normalization focuses on the weights and is independent of the concept of batch. Like Path-SGD, our goal is to obtain a balanced network ensuring a good back-propagation of the gradients, but our method explicitly re-balances the network using an iterative algorithm instead of using an implicit regularizer. Moreover, ENorm can be readily adapted to the convolutional case whereas Neyshabur et al. ( 2015) restrict themselves to the fully-connected case. In addition, the theoretical computational complexity of our method is much lower than the overhead introduced by BN or GN (see Section 4.5). Besides, WN parametrizes the weights in a polar-like manner, 𝑤 = 𝑔 × 𝑣/|𝑣|, where 𝑔 is a scalar and 𝑣 are the weights, thus it does not balance the network but individually scales each layer.

Equi-normalization

We first define Equi-normalization in the context of simple feed forward networks that consist of two operators: linear layers and ReLUs. The algorithm is inspired by Sinkhorn-Knopp and is designed to balance the energy of a network, i.e., the euclidian 𝑝-norm of its weights, while preserving its function. As shown in Theorem 4.3.1, the algorithm converges to a unique canonical network parametrization that minimizes the euclidian 𝑝-norm of its weights among equivalent networks. When not ambiguous, we may denote by network a parametrization 𝜃 of a given network architecture 𝐺.

Notation and Definitions

As defined in the Notations, we consider a ReLU network with 𝐿 linear layers, taking as input a row vector 𝑥 ∈ R 𝑁 0 . For the sake of exposition, we omit a bias term at this stage. We rely on the layer definition of rescalings (see 2.2.2) since it is closer to the implementation of the proposed algorithm. Recall that rescaling equivalence implies functional equivalence as shown in Proposition A.1.1. As detailed in Chapter 3, a functional equivalence class is not entirely described by rescaling operations. For example, permutations of neurons inside a layer also preserve functional equivalence, but do not change the gradient.

In what follows our objective is to seek a canonical parameter vector that is rescaling equivalent to a given parameter vector. The same objective under a functional equivalence constraint is beyond the scope of this Chapter, as there exist degenerate cases where functional equivalence does not imply rescaling equivalence, even up to permutations, as investigated in Chapter 3.

Objective Function: Canonical Representation

Given a network parameterization 𝜃 and 𝑝 > 0, we define the global 𝑝-norm of its weights as 𝐿 𝑝 (𝜃) = ∑ 𝐿 ℓ=1 ‖𝑊 (ℓ) ‖ 𝑝 𝑝 , where 𝑊 (ℓ) ∈ R 𝑁 ℓ-1 ×𝑁 ℓ . We are interested in minimizing 𝐿 𝑝 inside an equivalence class of neural networks in order to exhibit a unique canonical element per equivalence class. Recalling that 𝑁 ℓ denotes all the neurons of layer ℓ ∈ 0, 𝐿 , we denote the rescaling coefficients as 𝑑 ℓ ∈ [0, +∞] 𝑁 ℓ or as diagonal matrices 1 𝐷 (ℓ) = Diag(𝑑 ℓ ) ∈ 𝒟(|𝑁 ℓ |) for ℓ ∈ 1, 𝐿 -1 . Fixing the weights 𝑊 (ℓ) , we refer to

( 𝐷 (ℓ-1)
) -1 𝑊 (ℓ) 𝐷 (ℓ) as the rescaled weights, and seek to minimize their euclidian 𝑝-norm as a function of the rescaling coefficients:

𝜙(𝑑) = 𝐿 ∑ ℓ=1 ⃦ ⃦ ⃦ ⃦ ( 𝐷 (ℓ-1) ) -1 𝑊 (ℓ) 𝐷 (ℓ) ⃦ ⃦ ⃦ ⃦ 𝑝 𝑝 .
(4.1)

Coordinate Descent: ENorm Algorithm

We formalize the ENorm algorithm using the framework of block coordinate descent. We denote 2 by 𝑊 [:, 𝑗] (resp. 𝑊 (ℓ) [𝑖, :]) the 𝑗 th column (resp. 𝑖 th row) of a matrix 𝑊 (ℓ) .

In what follows we assume that 𝜃 is admissible 3 . Note that this is equivalent to saying that all the rows and columns of the hidden weight matrices 𝑊 (ℓ) are non-zero, as well as the columns (resp. rows) or 𝑊 1 (resp. 𝑊 𝑞 ). ENorm generates a sequence of rescalings 𝑑 (𝑟) by successively considering all the hidden layers ℓ ∈ 1, 𝐿 -1 to re-balance them as explained below.

Algorithm 1: Pseudo-code of Equi-normalization Input: Current layer weights 𝑊 (1) , . . . , 𝑊 (𝐿) , number of cycles 𝑇 , norm 𝑝 Output: Balanced layer weights // Perform T ENorm cycles for 𝑡 = 1 . . . 𝑇 do // 𝐷 (0) and 𝐷 (𝐿) are never updated, see Definition 2.2.2 for ℓ = 1 . . . 𝐿 -

1 do 𝐿[𝑗] ←- ⃦ ⃦ ⃦𝑊 (ℓ) [:, 𝑗] ⃦ ⃦ ⃦ 𝑝 for all 𝑗 ∈ R 𝑁 ℓ 𝑅[𝑖] ←- ⃦ ⃦ ⃦𝑊 (ℓ+1) [𝑖, :] ⃦ ⃦ ⃦ 𝑝 for all 𝑖 ∈ R 𝑁 ℓ 𝐷 (ℓ) ←-Diag √ 𝑅/𝐿 𝑊 (ℓ) ←-𝑊 (ℓ) 𝐷 (ℓ) 𝑊 (ℓ+1) ←- ( 𝐷 (ℓ) ) -1 𝑊 (ℓ+1)
(1) Initialization. Define 𝑑 (0) = (1, . . . , 1).

(2) Iteration. At iteration 𝑟, set ℓ such that ℓ -1 ≡ 𝑟 mod 𝐿 -1 and define

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ 𝑑 (𝑟+1) ℓ ′ = 𝑑 (𝑟) ℓ ′ if ℓ ′ ̸ = ℓ 𝑑 (𝑟+1) ℓ = argmin 𝑡∈[0,+∞] 𝑁 ℓ 𝜙 ( 𝑑 (𝑟) 1 , . . . , 𝑑 (𝑟) ℓ-1 , 𝑡, 𝑑 (𝑟) ℓ+1 , . . . , 𝑑 (𝑟) 𝐿-1 ) .
Denoting 𝑢𝑣 the coordinate-wise product of two vectors and 𝑢/𝑣 for the division,

𝑑 (𝑟+1) ℓ [𝑖] = ⎯ ⎸ ⎸ ⎸ ⎸ ⎷ ⃦ ⃦ ⃦𝑊 ℓ+1 [𝑖, :]𝑑 (𝑟) ℓ+1 ⃦ ⃦ ⃦ 𝑝 ⃦ ⃦ ⃦ 𝑊 ℓ [:, 𝑖]/𝑑 (𝑟) ℓ-1 ⃦ ⃦ ⃦ 𝑝 . (4.2)
Algorithm and pseudo-code. Algorithm 1 gives the pseudo-code of ENorm. By convention, one ENorm cycle balances the entire network once from ℓ = 1 to ℓ = 𝐿-1. See Appendix B.1 for illustrations showing the effect of ENorm on network's weights.

Convergence

We now state our main convergence result for Equi-normalization. The proof relies on a coordinate descent Theorem by [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF] and can be found in Appendix B.2.

Theorem 4.3.1. Let 𝑝 > 0 and (𝑑 (𝑟) ) 𝑟∈N be the sequence of rescaling coefficients generated by ENorm from the starting point 𝑑 (0) as described in Section 4.3.3. We assume that the considered parameterization 𝜃 is admissible. Then,

(1) Convergence. The sequence of rescaling coefficients 𝑑 (𝑟) converges to 𝑑 * as 𝑟 → +∞. As a consequence, the sequence of rescaled weights also converges;

(2) Minimum norm. The rescaled weights after convergence minimize the global 𝑝-norm among all rescaling equivalent weights;

(3) Uniqueness. The minimum 𝑑 * does not depend on the starting point 𝑑 (0) .

Gradients & Biases

From now on, we denote the rescaled quantities (weights, biases, activations) with a tilde. In the presence of biases, as explained in Definition 2.2.2, we define matched rescaling equivalent biases ̃ 𝑏 (ℓ) = 𝑏 (ℓ) 𝐷 (ℓ) . Thus, we have, for all ℓ ∈ 0, 𝐿 ,

 𝑦 (ℓ) = 𝑦 (ℓ) 𝐷 (ℓ) (4.3)
We also compute the effect of applying ENorm on the gradients in Appendix B.1.1.

Asymmetric Scaling

Equi-normalization is easily adapted to introduce a depth-wise penalty on each layer. We consider the weighted loss 𝐿 𝑝,(𝑐 1 ,...,𝑐 𝐿 ) (𝜃) = ∑ 𝐿 ℓ=1 𝑐 ℓ ‖𝑊 (ℓ) ‖ 𝑝 . This amounts to modifying the rescaling coefficients as

𝑑 (𝑟+1) ℓ [𝑖] ← 𝑑 (𝑟+1) ℓ [𝑖] (𝑐 ℓ+1 /𝑐 ℓ ) 1/2𝑝 .
(4.4)

In Section 4.6, we explore two natural ways of defining 𝑐 ℓ : 𝑐 ℓ = 𝑐 𝑝(𝐿-ℓ) (uniform) and

𝑐 ℓ = 1/(𝑁 ℓ-1 𝑁 ℓ ) (adaptive).
In the uniform setup, we penalize layers exponentially according to their depth: for instance, values of 𝑐 larger than 1 increase the magnitude of the weights at the end of the network. In the adaptive setup, the loss is weighted by the size of the matrices.

Extension to CNNs

We now extend ENorm to CNNs, by focusing on the typical ResNet architecture. We briefly detail how we adapt ENorm to convolutional or max-pooling layers, and then how to update an elementary block with a skip-connection. We refer the reader to

Convolutional Layers

Figure 4-2 explains how to rescale two consecutive convolutional layers. As detailed in Appendix B.3, this is done by first properly reshaping the filters to 2D matrices, then performing the previously described rescaling on these matrices, and then reshaping the matrices back to convolutional filters. This matched rescaling does preserve the function implemented by the composition of the two layers, whether they are interleaved with a ReLU or not. It can be applied to any two consecutive convolutional layers with various stride and padding parameters. Note that when the kernel size is 1 in both layers, we recover the fully-connected case of Figure 4-1.

Max-Pooling

The MaxPool layer operates per channel by computing the maximum within a fixedsize kernel. We use Equation (4.3) to the convolutional case where the rescaling matrix 𝐷 (ℓ) is applied to the channel dimension of the activations 𝑦 (ℓ-1) . Then, max

( ̃ 𝑦 (ℓ) ) = max ( 𝑦 (ℓ) 𝐷 (ℓ) ) = max ( 𝑦 (ℓ)
)

𝐷 (ℓ) . (4.5) Thus, the activations before and after the MaxPool layer have the same scaling and the functional equivalence is preserved.

Skip Connections

We now consider an elementary block of a ResNet-18 architecture as depicted in Figure 4-3. In order to maintain functional equivalence, we only consider ResNet architectures of type C as defined in (He et al., 2015a), where all shortcuts are learned 1 × 1 convolutions. As detailed in Appendix B.3, rescaling two consecutive blocks requires (a) to define the structure of the rescaling process, i.e. where to insert the rescaling coefficients and (b) a formula for computing these rescaling coefficients.

Training with Equi-normalization & SGD

ENorm & SGD. As detailed in Algorithm 2, we balance the network periodically after updating the gradients. By design, this does not change the function implemented by the network but will yield different gradients in the next SGD iteration. Because this re-parameterization performs a jump in the parameter space, we update the momentum as described in Appendix B.1.1 and the same matrices 𝐷 (ℓ) as these used for the weights. The number of ENorm cycles after each SGD step is an hyperparameter and by default we perform one ENorm cycle after each SGD step. In

MNIST Autoencoder

Training. Input data is normalized by subtracting the mean and dividing by standard deviation. The encoder has the structure FC(784, 1000)-ReLU-FC(1000, 500)-ReLU-FC(500, 250)-ReLU-FC(250, 30) and the decoder has the symmetric structure. We use He's initialization for the weights. We select the learning rate in {0.001, 0.01, 0.1} and decay it linearly to zero. We use a batch size of 256 and SGD with no momentum and a weight decay of 0.001. For path-SGD, our implementation closely follows the original paper [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF] and we set the weight decay to zero. For GN, we cross-validate the number of groups among {5, 10, 20, 50}. For WN, we use BN as well as a greedy layer-wise initialization as in the original paper.

Results. While ENorm alone obtains competitive results compared to BN and GN, ENorm + BN outperforms all other methods, including WN + BN. Note that here ENorm refers to Enorm using the adaptive 𝑐 parameter as described in Subsection 4.3.6, whereas for ENorm + BN we use the uniform setup with 𝑐 = 1. We perform a parameter study for different values and setups of the asymmetric scaling (uniform and adaptive) in Appendix B.5.2. Without BN, the adaptive setup outperforms all other setups, which may be due to the strong bottleneck structure of the network. With BN, the dynamic is different and the results are much less sensitive to the values of 𝑐. Results without any normalization and with Path-SGD are not displayed because of their poor performance.

CIFAR-10 Fully Connected

Training. We first experiment with a basic fully-connected architecture that takes as input the flattened image of size 3072. Input data is normalized by subtracting mean and dividing by standard deviation independently for each channel. The first linear layer is of size 3072 × 500. We then consider 𝑝 layers 500 × 500, 𝑝 being an architecture parameter for the sake of the analysis. The last classification is of size 500 × 10. The weights are initialized with He's scheme. We train for 60 epochs using SGD with no momentum, a batch size of 256 and weight decay of 10 -3 . Cross validation is used to pick an initial learning rate in {0.0005, 0.001, 0.005, 0.01, 0.05, 0.1}. Path-SGD, GN and WN are learned as detailed in Section 4.6.1. All results are the average test accuracies over 5 training runs. Results. ENorm alone outperforms both BN and GN for any depth of the network. ENorm + BN outperforms all other methods, including WN + BN, by a good margin for more than 𝑝 = 11 intermediate layers. Note that here ENorm as well as ENorm + BN refers to ENorm in the uniform setup with 𝑐 = 1.2. The results of the parameter study for different values and setups of the asymmetric scaling are similar to these of the MNIST setup, see Appendix B.5.2.

CIFAR-10 Fully Convolutional

Training. We use the CIFAR-NV architecture as described by [START_REF] Gitman | Comparison of batch normalization and weight normalization algorithms for the large-scale image classification[END_REF]. Images are normalized by subtracting mean and dividing by standard deviation independently for each channel. During training, we use 28 × 28 random crops and randomly flip the image horizontally. At test time, we use 28 × 28 center crops. We split the train set into one training set (40,000 images) and one validation set (10,000 images). We train for 128 epochs using SGD and an initial learning rate cross-validated on a held-out set among {0.01, 0.05, 0.1}, along with a weight decay of 0.001. The learning rate is then decayed linearly to 10 -4 . We use a momentum of 0.9. The weights are initialized with He's scheme. In order to stabilize the training, we employ a BatchNorm layer at the end of the network after the FC layer for the Baseline and ENorm cases. For GN we cross-validate the number of groups among {4, 8, 16, 32, 64}. All results are the average test accuracies over 5 training runs.

Results. See

ImageNet

This dataset contains 1.3M training images and 50,000 validation images split into 1000 classes. We use the ResNet-18 model with type-C learnt skip connections as described in Section 4.4.

Training. Our experimental setup closely follows that of GN [START_REF] Wu | Group normalization[END_REF]. We train on 8 GPUs and compute the batch mean and standard deviation per GPU when evaluating BN. We use the Kaiming initialization for the weights (He et al., 2015c) and the standard data augmentation scheme of ?. We train our models for 90 epochs using SGD with a momentum of 0.9. We adopt the linear scaling rule for the learning rate [START_REF] Goyal | Accurate, large minibatch sgd: Training imagenet in 1 hour[END_REF] and set the initial learning rate to 0.1𝐵/256 where the batch size 𝐵 is set to 32, 64, 128, or 256. As smaller batches lead to more iterations per epoch, we adopt a similar rule and adopt a weight decay of 𝑤 = 10 -4 for 𝐵 = 128 and 256, 𝑤 = 10 -4.5 for 𝐵 = 64 and 𝑤 = 10 -5 for 𝐵 = 32. We decay the learning rate quadratically [START_REF] Gitman | Comparison of batch normalization and weight normalization algorithms for the large-scale image classification[END_REF] to 10 -5 and report the median error rate on the final 5 epochs. When using GN, we set the number of groups 𝐺 to 32 and did not cross-validate this value as prior work [START_REF] Wu | Group normalization[END_REF] reports little impact when varying 𝐺 from 2 to 64. In order for the training to be stable and faster, we added a BatchNorm at the end of the network after the FC layer for the Baseline and ENorm cases. The batch mean and variance for this additional BN are shared across GPUs. Note that the activation size at this stage of the network is 𝐵 × 1000, which is a negligible overhead (see Table 4.1).

Results. We compare the Top 1 accuracy on a ResNet-18 when using no normalization scheme (Baseline), when using BN, GN and ENorm (our method). In both the Baseline and ENorm settings, we add a BN at the end of the network as described in Section 4.6.3. The results are reported in

Limitations

We applied ENorm to a deeper (ResNet-50), but obtained unsatisfactory results. We observed that learnt skip-connections, even initialized to identity, make it harder to train without BN, even with careful layer-wise initialization or learning rate warmup. This would require further investigation.

Conclusion

We presented Equi-normalization, an iterative method that balances the energy of the weights of a network while preserving the function it implements. ENorm provably converges towards a unique equivalent network that minimizes the 𝑝-norm of its weights and it can be applied to modern CNN architectures. Using ENorm during training adds a much smaller computational overhead than BN or GN and leads to competitive performances in the FC case as well as in the convolutional case.

While optimizing an unbalanced network is hard [START_REF] Behnam Neyshabur | Path-SGD: Pathnormalized optimization in deep neural networks[END_REF], the criterion we optimize to derive ENorm-minimizing the 𝐿 2 norm among the equivalence class -is likely not optimal regarding convergence or training properties. This limitation suggests that further research is required in this direction.

Chapter 5

Compressing Networks with Iterative Product Quantization

In this Chapter, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5 MB (20× compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26× factor.

Introduction

There is a growing need for compressing the best convolutional networks (ConvNets) to support embedded devices for applications like robotics and virtual or augmented reality. Indeed, the performance of ConvNets on image classification has steadily improved since the introduction of AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. This progress has been fueled by deeper and richer architectures such as the ResNets (He et al., 2015a) and their variants ResNeXts [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] or DenseNets (Huang et al., 2017a). these models particularly benefit from the recent progress made with weak supervision [START_REF] Yalniz | Billionscale semi-supervised learning for image classification[END_REF][START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF]. Compression of ConvNets has been an active research topic in the recent years, leading to networks with a 71% top-1 accuracy on ImageNet object classification that fit in 1 MB (Wang et al., 2018a).

in-domain

Figure 5-1: We approximate a binary classifier 𝜙 that labels images as dogs or cats by quantizing its weights. Standard method: quantizing 𝜙 with the standard objective function (5.1) promotes a classifier ̂ 𝜙 standard that tries to approximate 𝜙 over the entire input space and can thus perform badly for in-domain inputs. Our method: quantizing 𝜙 with our objective function (5.2) promotes a classifier ̂ 𝜙 activations that performs well for in-domain inputs. Images lying in the hatched area of the input space are correctly classified by 𝜙 activations but incorrectly by 𝜙 standard .

In this work, we propose a compression method particularly adapted to ResNetlike architectures. Our approach takes advantage of the high correlation in the convolutions by the use of a structured quantization algorithm, Product Quantization (PQ) [START_REF] Jegou | Product quantization for nearest neighbor search[END_REF]. More precisely, we exploit the spatial redundancy of information inherent to standard convolution filters [START_REF] Emily L Denton | Exploiting linear structure within convolutional networks for efficient evaluation[END_REF]. Besides reducing the memory footprint, we produce compressed networks allowing efficient inference on CPU, as opposed to entropy decoders (Han et al., 2016b).

Our approach departs from traditional scalar quantizers (Han et al., 2016b) and vector quantizers [START_REF] Gong | Compressing deep convolutional networks using vector quantization[END_REF][START_REF] Miguel | Model compression as constrained optimization, with application to neural nets[END_REF]) by focusing on the accuracy of the activations rather than the weights. This is achieved by leveraging a weighted 𝑘-means technique. To our knowledge this strategy is novel in this context. The closest work we are aware of is the one by [START_REF] Choi | Towards the limit of network quantization[END_REF], but the authors use a different objective (their weighted term is derived from second-order information) along with a different quantization technique (scalar quantization). Our method targets a better in-domain reconstruction, as depicted by Figure 5-1.

Finally, we compress the network sequentially to account for the dependency of our method to the activations at each layer. To prevent the accumulation of errors across layers, we guide this compression with the activations of the uncompressed network on unlabelled data: training by distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] allows for both an efficient layer-by-layer compression procedure and a global fine-tuning of the codewords. Thus, we only need a set of unlabelled images to adjust the codewords. As opposed to recent works by Mishra and Marr (2017), Lopes et al. (2017), our distillation scheme is sequential and the underlying compression method is different. Similarly, [START_REF] Wu | Quantized convolutional neural networks for mobile devices[END_REF] use use Vector Quantization (VQ) instead PQ, do not finetune the learned codewords and do not compress the classifier's weights.

We show that applying our approach to the semi-supervised ResNet-50 of [START_REF] Yalniz | Billionscale semi-supervised learning for image classification[END_REF] leads to a 5 MB memory footprint and a 76.1% top-1 accuracy on Ima-geNet object classification (hence 20× compression vs. the original model). Moreover, our approach generalizes to other tasks such as image detection. As shown in Section 5.4.3, we compress a Mask R-CNN (He et al., 2017a) with a size budget around 6 MB (26× compression factor) while maintaining a competitive performance.

Related work

Here, we review the works closest to ours and refer the reader to the Related Work section 2.3 for a more extensive discussion.

Low-precision Training.

Since early works like these of [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF], researchers have developed various approaches to train networks with low precision weights. these approaches include training with binary or ternary weights [START_REF] Shayer | Learning discrete weights using the local reparameterization trick[END_REF][START_REF] Zhu | Trained ternary quantization[END_REF][START_REF] Li | Ternary weight networks[END_REF][START_REF] Rastegari | Xnornet: Imagenet classification using binary convolutional neural networks[END_REF][START_REF] Mark | Training wide residual networks for deployment using a single bit for each weight[END_REF], learning a combination of binary bases (Lin et al., 2017b) and quantizing the activations [START_REF] Zhou | Dorefanet: Training low bitwidth convolutional neural networks with low bitwidth gradients[END_REF][START_REF] Zhou | Incremental network quantization: Towards lossless CNNs with low-precision weights[END_REF]Mishra et al., 2017). Some of these methods assume the possibility to employ specialized hardware that speed up inference and improve power efficiency by replacing most arithmetic operations with bit-wise operations. However, the back-propagation has to be adapted to the case where the weights are discrete using accumulation or projection techniques and binary networks suffer from a significant drop in accuracy despite noticeable progress.

Quantization. Vector Quantization (VQ) and Product Quantization (PQ) have been extensively studied in the context of nearest-neighbor search [START_REF] Jegou | Product quantization for nearest neighbor search[END_REF][START_REF] Ge | Optimized product quantization[END_REF][START_REF] Norouzi | Cartesian k-means[END_REF]. The idea is to decompose the original high-dimensional space into a cartesian product of subspaces that are quantized separately with a joint codebook. To our knowledge, [START_REF] Gong | Compressing deep convolutional networks using vector quantization[END_REF] were the first to introduce these stronger quantizers for neural network quantization, followed by

Our approach

In this section, we describe our strategy for network compression and we show how to extend our approach to quantize a modern ConvNet architecture. The specificity of our approach is that it aims at a small reconstruction error for the outputs of the layer rather than the layer weights themselves. We first describe how we quantize a single fully connected and convolutional layer. Then we describe how we quantize a full pre-trained network and finetune it. We call the proposed algorithm iPQ for Iterative Product Quantization.

Quantization of a Fully-connected Layer

We consider a fully-connected layer with weights 𝑊 ∈ R 𝐶 in ×𝐶out and, without loss of generality, we omit the bias since it does not impact reconstruction error.

Product Quantization (PQ).

When applying the PQ algorithm to the columns of 𝑊 , we evenly split each column into 𝑚 contiguous subvectors and learn a codebook on the resulting 𝑚𝐶 out subvectors. Then, a column of 𝑊 is quantized by mapping each of its subvector to its nearest codeword in the codebook. For simplicity, we assume that 𝐶 in is a multiple of 𝑚. Hence, all the subvectors have the same dimension 𝑑 = 𝐶 in /𝑚.

More formally, the codebook 𝒞 = {𝑐 1 , . . . , 𝑐 𝑘 } contains 𝑘 codewords of dimension 𝑑. Any column 𝑤 𝑗 of 𝑊 is mapped to its quantized version 𝑞(𝑤 𝑗 ) = (𝑐 𝑖 1 , . . . , 𝑐 𝑖𝑚 ) where 𝑖 1 denotes the index of the codeword assigned to the first subvector of 𝑤 𝑗 , and so forth. The codebook is then learned by minimizing the following objective:

‖𝑊 -̂ 𝑊 ‖ 2 2 = ∑ 𝑗 ‖𝑤 𝑗 -𝑞(𝑤 𝑗 )‖ 2 2 , (5.1)
where ̂ 𝑊 denotes the quantized weights. This objective can be efficiently minimized with 𝑘-means. When 𝑚 is set to 1, PQ is equivalent to vector quantization (VQ) and when 𝑚 is equal to 𝐶 in , it is the scalar 𝑘-means algorithm. The main benefit of PQ is its expressivity: each column 𝑤 𝑗 is mapped to a vector in the product 𝒞 = 𝒞 × • • • × 𝒞, thus PQ generates an implicit codebook of size 𝑘 𝑚 .

Our algorithm. PQ quantizes the weight matrix of the fully-connected layer. However, in practice, we are interested in preserving the output of the layer, not its weights. This is illustrated in the case of a non-linear classifier in Figure 5-1: preserving the weights a layer does not necessarily guarantee preserving its output. In other words, the Frobenius approximation of the weights of a layer is not guaranteed to be the best approximation of the output over some arbitrary domain (in particular for in-domain inputs). We thus propose an alternative to PQ that directly minimizes the reconstruction error on the output activations obtained by applying the layer to in-domain inputs. More precisely, given a batch of 𝐵 input activations 𝑥 ∈ R 𝐵×𝐶 in , we are interested in learning a codebook 𝒞 that minimizes the difference between the output activations and their reconstructions:

‖𝑦 -̂ 𝑦‖ 2 2 = ∑ 𝑗 ‖𝑥(𝑤 𝑗 -𝑞(𝑤 𝑗 ))‖ 2 2 , (5.2)
where 𝑦 = 𝑥𝑊 is the output and ̂ 𝑦 = 𝑥 ̂ 𝑊 its reconstruction. Our objective is a re-weighting of the objective in Equation (5.1). We can thus learn our codebook with a weighted 𝑘-means algorithm. First, we unroll 𝑥 of size 𝐵 × 𝐶 in into ̃ 𝑥 of size (𝐵 × 𝑚) × 𝑑 i.e. we split each row of 𝑥 into 𝑚 subvectors of size 𝑑 and stack these subvectors. Next, we adapt the EM algorithm as follows.

(1) E-step (cluster assignment). Recall that every column 𝑤 𝑗 is divided into 𝑚 subvectors of dimension 𝑑. Each subvector 𝑣 is assigned to the codeword

𝑐 𝑗 = argmin 𝑐∈𝒞 ‖̃ 𝑥(𝑐 -𝑣)‖ 2 2 .
(5.3)

This step is performed by exhaustive exploration. Our implementation relies on broadcasting to be computationally efficient.

(2) M-step (codeword update). Let 𝑐 ∈ 𝒞. We denote (𝑣 𝑝 ) 𝑝∈𝐼𝑐 the subvectors that are currently assigned to 𝑐. Then, we update 𝑐 ← 𝑐 ⋆ , where

𝑐 ⋆ = argmin 𝑐∈R 𝑑 ∑ 𝑝∈𝐼𝑐 ‖̃ 𝑥(𝑐 -𝑣 𝑝 )‖ 2 2 .
(5.4)

This step explicitly computes the solution of the least-squares problem 1 . Our implementation performs the computation of the pseudo-inverse of ̃ 𝑥 before alternating between the Expectation and Minimization steps as it does not depend on the learned codebook 𝒞.

We initialize the codebook 𝒞 by uniformly sampling 𝑘 vectors among these we wish to quantize. After performing the E-step, some clusters may be empty. To resolve this 1 Denoting ̃ 𝑥 + the Moore-Penrose pseudo-inverse of ̃ 𝑥, we obtain

𝑐 * = 1 |𝐼𝑐| ̃ 𝑥 + ̃ 𝑥 ( ∑ 𝑝∈𝐼𝑐 𝑣 𝑝 )
issue, we iteratively perform the following additional steps for each empty cluster of index 𝑖.

(1) Find codeword 𝑐 0 corresponding to the most populated cluster ; (2) define new codewords 𝑐 ′ 0 = 𝑐 0 + 𝑒 and 𝑐 ′ 𝑖 = 𝑐 0 -𝑒, where 𝑒 ∼ 𝒩 (0, 𝜀) and ( 3) perform again the E-step. We proceed to the M-step after all the empty clusters are resolved. We set 𝜀 = 1e-8 and we observe that its generally takes less than 1 or 2 E-M iterations to resolve all the empty clusters. Note that the quality of the resulting compression is sensitive to the choice of 𝑥.

Convolutional Layers

Despite being presented in the case of a fully-connected layer, our approach works on any set of vectors. As a consequence, our approach can be applied to a convolutional layer if we split the associated 4D weight matrix into a set of vectors. There are many ways to split a 4D matrix in a set of vectors and we are aiming for one that maximizes the correlation between the vectors since vector quantization based methods work the best when the vectors are highly correlated.

Given a convolutional layer, we have 𝐶 out filters of size 𝐾 × 𝐾 × 𝐶 in , leading to an overall 4D weight matrix 𝑊 ∈ R 𝐶out×𝐶 in ×𝐾×𝐾 . The dimensions along the output and input coordinate have no particular reason to be correlated. On the other hand, the spatial dimensions related to the filter size are by nature very correlated: nearby patches or pixels likely share information. As depicted in Figure 5-2, we thus reshape the weight matrix in a way that lead to spatially coherent quantization. More precisely, we quantize 𝑊 spatially into subvectors of size 𝑑 = 𝐾×𝐾 using the following procedure. We first reshape 𝑊 into a 2D matrix of size (𝐶 in ×𝐾 ×𝐾)×𝐶 out . Column 𝑗 of the reshaped matrix 𝑊 r corresponds to the 𝑗 th filter of 𝑊 and is divided into 𝐶 in subvectors of size 𝐾 × 𝐾. Similarly, we reshape the input activations 𝑥 accordingly to 𝑥 r so that reshaping back the matrix 𝑥 r 𝑊 r yields the same result as 𝑥 * 𝑊 . In other words, we adopt a dual approach to the one using bi-level Toeplitz matrices to represent the weights. Then, we apply our method exposed in Section 5.3.1 to quantize each column of 𝑊 r into 𝑚 = 𝐶 in subvectors of size 𝑑 = 𝐾 × 𝐾 with 𝑘 codewords, using 𝑥 r as input activations in (5.2). We also quantize with larger subvectors, for example subvectors of size 𝑑 = 2 × 𝐾 × 𝐾, see Section 5.4 for details.

In our implementation, we adapt the reshaping of 𝑊 and 𝑥 to various types of convolutions and refer the reader to the code for more details.

Network Quantization

In this section, we describe our approach for quantizing a neural network. We quantize the network sequentially starting from the lowest layer to the highest layer, and guide the compression of the student network by the non-compressed teacher network.

Learning the codebook. We recover the current input activations of the layer, i.e. the input activations obtained by forwarding a batch of images through the quantized lower layers, and we quantize the current layer using these activations. Experimentally, we observed a drift in both the reconstruction and classification errors when using the activations of the non-compressed network instead.

Finetuning the codebook. We finetune the codewords by distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] using the non-compressed network as the teacher network and the compressed network (up to the current layer) as the student network. Denoting 𝑦 t (resp. 𝑦 s ) the output probabilities of the teacher (resp. student) network, the loss we optimize is the Kullback-Leibler divergence ℒ = KL(𝑦 s , 𝑦 t ). Finetuning on codewords is done by averaging the gradients of each subvector assigned to a given codeword. More formally, after the quantization step, we fix the assignments once for all. Then, denoting (𝑏 𝑝 ) 𝑝∈𝐼𝑐 the subvectors that are assigned to codeword 𝑐, we perform the SGD update with a learning rate 𝜂

𝑐 ← 𝑐 -𝜂 1 |𝐼 𝑐 | ∑ 𝑝∈𝐼𝑐 𝜕ℒ 𝜕𝑏 𝑝 .
(5.5)

Experimentally, we find the approach to perform better than finetuning on the target of the images as demonstrated in Table 5.3. Moreover, this approach does not require any labelled data as it relies on distillation.

Global Finetuning

In a final step, we globally finetune the codebooks of all the layers to reduce any residual drifts and we update the running statistics of the BatchNorm layers: We empirically find it beneficial to finetune all the centroids after the whole network is quantized. The finetuning procedure is exactly the same as described in Section 5.3.3, except that we additionally switch the BatchNorms to the training mode, meaning that the learnt coefficients are still fixed but that the batch statistics (running mean and variance) are still being updated with the standard moving average procedure.

We perform the global finetuning using the standard ImageNet training set for 9 epochs with an initial learning rate of 0.01, a weight decay of 10 -4 and a momentum of 0.9. The learning rate is decayed by a factor 10 every 30 epochs. As demonstrated in the ablation study in Table 5.3, finetuning on the true labels performs worse than finetuning by distillation. A possible explanation is that the supervision signal coming from the teacher network is richer than the one-hot vector used traditionally.

Experiments

Experimental Setup

We quantize vanilla ResNet-18 and ResNet-50 architectures pretrained on the Im-ageNet dataset [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. Unless explicit mention of the contrary, the pretrained models are taken from the PyTorch model zoo2 . We run our method on a 16 GB Volta V100 GPU. Quantizing a ResNet-50 with our method (including all finetuning steps) takes about one day on 1 GPU. We detail our experimental setup below. Our code and the compressed models are open-sourced.

Compression Regimes. We explore a large block sizes (resp. small block sizes) compression regime by setting the subvector size of 3×3 convolutions to 𝑑 = 9 (resp. 𝑑 = 18) and the subvector size of pointwise convolutions to 𝑑 = 4 (resp. 𝑑 = 8). For ResNet-18, the block size of pointwise convolutions is always equal to 4. The number of codewords or centroids is set to 𝑘 ∈ {256, 512, 1024, 2048} for each compression regime. We clamp the number of centroids to min(𝑘, 𝐶 out × 𝑚/4) for stability. For instance, the first layer of the first stage of the ResNet-50 has size 64× 64× 1 ×1, thus we always use 𝑘 = 128 centroids with a block size 𝑑 = 8. For a given number of centroids 𝑘, small blocks lead to a lower compression than large blocks.

Sampling the Input Activations. Before quantizing each layer, we randomly sample a batch of 1024 training images to obtain the input activations of the current layer and reshape it as described in Section 5.3.2. Before each iteration (E+M step) of our method, we randomly sample 10, 000 rows from these reshaped activations.

Hyperparameters. We quantize each layer while performing 100 steps of our method (sufficient for convergence in practice). We finetune the centroids of each layer on the standard ImageNet training set during 2,500 iterations with a batch size of 128 (resp 64) for the with a learning rate of 0.01, a weight decay of 10 -4 and a momentum of 0.9. For accuracy and memory reasons, the classifier is always quantized with a block size 𝑑 = 4 and 𝑘 = 2048 (resp. 𝑘 = 1024) centroids for the . Moreover, the first convolutional layer of size 7 × 7 is not quantized, as it represents less than 0.1% (resp., 0.05%) of the weights of a .

Metrics. We focus on the tradeoff between accuracy and memory. The accuracy is the top-1 error on the standard validation set of ImageNet. The memory footprint is calculated as the indexing cost plus the overhead of storing the centroids in float16. As an example, quantizing a layer of size 128 × 128 × 3 × 3 with 𝑘 = 256 centroids (1 byte per subvector) and a block size of 𝑑 = 9 leads to an indexing cost of 16 kB for 𝑚 = 16, 384 blocks plus the cost of storing the centroids of 4.5 kB.

Image Classification Results

We report below the results of our method applied to various ResNet models. First, we compare our method with the state of the art on the standard ResNet-18 and ResNet-50 architecture. Next, we show the potential of our approach on a competitive ResNet-50. Finally, an ablation study validates the pertinence of our method.

Vanilla ResNet-18 and ResNet-50. We evaluate our method on the ImageNet benchmark for the following methods: Trained Ternary Quantization (TTQ) [START_REF] Zhu | Trained ternary quantization[END_REF], LR-Net [START_REF] Shayer | Learning discrete weights using the local reparameterization trick[END_REF], ABC-Net (Lin et al., 2017b), Binary Weight Network (XNOR-Net or BWN) [START_REF] Rastegari | Xnornet: Imagenet classification using binary convolutional neural networks[END_REF], Deep Compression (DC) (Han et al., 2016b) and Hardware-Aware Automated Quantization (HAQ) (Wang et al., 2018a). We report the accuracies and compression factors in the original papers and/or in the two surveys [START_REF] Guo | A survey on methods and theories of quantized neural networks[END_REF][START_REF] Cheng | A survey of model compression and acceleration for deep neural networks[END_REF] for a given architecture when the result is available. We do not compare our method to DoReFa-Net [START_REF] Zhou | Dorefanet: Training low bitwidth convolutional neural networks with low bitwidth gradients[END_REF] and WRPN (Mishra et al., 2017) as these approaches also use low-precision activations and hence get lower accuracies, e.g., 51.2% top-1 accuracy for a XNOR-Net with ResNet-18. The results are presented in Figure 5.4.2. For better readability, some results for our method are also displayed in Table 5.1. We report the average accuracy and standard deviation over 3 runs. Our method significantly outperforms state of the art papers for various operating points. For instance, for a ResNet-18, our method with large blocks and 𝑘 = 512 centroids reaches a larger accuracy than ABC-Net (𝑀 = 2) with a compression ratio that is 2x larger. Similarly, on the ResNet-50, our compressed model with 𝑘 = 256 centroids in the large blocks setup yields a comparable accuracy to DC (2 bits) with a compression ratio that is 2x larger.

The work by [START_REF] Tung | Deep neural network compression by in-parallel pruning-quantization[END_REF] is likely the only one that remains competitive with ours with a 6.8 MB network after compression, with a technique that prunes the network and therefore implicitly changes the architecture. The authors report the delta accuracy for which we have no direct comparable top-1 accuracy, but their method is arguably complementary to ours.

Semi-supervised ResNet-50. Recent works [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF][START_REF] Yalniz | Billionscale semi-supervised learning for image classification[END_REF] have demonstrated the possibility to leverage a large collection of unlabelled images to improve the accuracy. In particular, [START_REF] Yalniz | Billionscale semi-supervised learning for image classification[END_REF] use the publicly available YFCC-100M dataset [START_REF] Thomee | The new data and new challenges in multimedia research[END_REF] to train a ResNet-50 that reaches 79.3% top-1 accuracy on the standard validation set of ImageNet. In the following, we use this particular model and refer to it as semi-supervised ResNet-50. In the low Comparison for a Given Size Budget. To ensure a fair comparison, we compare our method for a given model size budget against the reference methods in Table 5.2. It should be noted that our method can further benefit from advances in semi-supervised learning to boosts the performance of the non-compressed and hence of the compressed network.

Ablation Study.

We perform an ablation study on the vanilla ResNet-18 to study the respective effects of quantizing using the activations and finetuning by distillation (here, finetuning refers both to the per-layer finetuning and to the global finetuning after the quantization described in Section 5.3). We refer to our method as Act + Distill. First, we still finetune by distillation but change the quantization: instead of quantizing using our method (see Equation (5.2)), we quantizing using the standard PQ algorithm and do not take the activations into account, see Equation (5.1). We refer to this method as No act + Distill. Second, we quantize using our method but perform a standard finetuning using the image labels (Act + Labels). The results are displayed in Table 5.3. Our approach consistently yields significantly better results. As a side note, quantizing all the layers of a ResNet-18 with the standard PQ algorithm and without any finetuning leads to top-1 accuracies below 25% for all operating points, which illustrates the drift in accuracy occurring when compressing deep networks with standard methods (as opposed to our method).

Image Detection Results

To demonstrate the generality of our method, we compress the Mask R-CNN architecture used for image detection in many real-life applications (He et al., 2017a). We compress the backbone (ResNet-50 FPN) in the small blocks compression regime and refer the reader to the open-sourced compressed model for the block sizes used in the various heads of the network. Results are displayed in

Conclusion

We presented a quantization method based on Product Quantization that gives state of the art results on ResNet architectures and that generalizes to other architectures such as Mask R-CNN. Our compression scheme does not require labeled data and the resulting models are byte-aligned, allowing for efficient inference on CPU.

A current limitation of our method is that it takes advantage of the natural spatial redundancy in the traditional 3×3 filters and performs worse on 1×1 filters (pointwise convolutions), where the redundancy, if any, is less observable. This makes iPQ less suited for mobile-efficient architectures such as MobileNets or EfficientNets (Sandler et al., 2018a;[START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF]. We developed a pre-conditioning technique called Quant-Noise that is explained in the next Chapter 6.

Chapter 6

Pre-conditioning Network Compression with Quant-Noise

As in Chapter 5, we tackle the problem of producing compact models, maximizing their accuracy for a given model size. To that end, we develop a pre-conditioning technique called Quant-Noise that injects a carefully chosen quantization noise when training the uncompressed network before compressing it with iPQ. As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classification. For example, applying our method to state-of-the-art Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB and 80.0% top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3 MB. We also show the potential of Quant-Noise for scalar int8 and int4 quantization over the standard Quantization Aware Training [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] procedure. Finally, we combine product and int8 scalar quantization to take advantage of (i) the speedup provided by int8 and the high compression ratio of iPQ.

Introduction

Many of the best performing neural network architectures in real-world applications have a large number of parameters. For example, the current standard machine translation architecture, Transformer [START_REF] Vaswani | Attention is all you need[END_REF], has layers that contain millions of parameters. Even models that are designed to jointly optimize the performance and the parameter efficiency, such as EfficientNets [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF], still require dozens to hundreds of megabytes, which limits their applications to domains like robotics or virtual assistants.

Figure 6-1: Quant-Noise trains models to be resilient to quantization by mimicking the effect of the quantization method during training time. This allows for extreme compression rates without much loss in accuracy on a variety of tasks and benchmarks.

Model compression schemes reduce the memory footprint of overparametrized models. Pruning (LeCun et al., 1990) and distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] remove parameters by reducing the number of network weights. In contrast, quantization focuses on reducing the bits per weight. This makes quantization particularly interesting when compressing models that have already been carefully optimized in terms of network architecture. Whereas deleting weights or whole hidden units will inevitably lead to a drop in performance, we demonstrate that quantizing the weights can be performed with little to no loss in accuracy.

Popular postprocessing quantization methods, like scalar quantization, replace the floating-point weights of a trained network by a lower-precision representation, like fixed-width integers [START_REF] Vanhoucke | Improving the speed of neural networks on cpus[END_REF]. These approaches achieve a good compression rate with the additional benefit of accelerating inference on supporting hardware. However, the errors made by these approximations accumulate in the computations operated during the forward pass, inducing a significant drop in performance as explained in Chapter 5.

A solution to address this drifting effect is to directly quantize the network during training. This raises two challenges. First, the discretization operators have a null gradient -the derivative with respect to the input is zero almost everywhere. This requires special workarounds to train a network with these operators. The second challenge that often comes with these workarounds is the discrepancy that appears between the train and test functions implemented by the network. Quantization Aware Training (QAT) [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] resolves these issues by quantizing all the weights during the forward and using a straight through estimator (STE) [START_REF] Bengio | Estimating or propagating gradients through stochastic neurons for conditional computation[END_REF] to compute the gradient. This works when the error introduced by STE is small, like with int8 quantization, but does not suffice in compression regimes where the approximation made by the compression is more severe.

In this work, we show that quantizing only a subset of weights instead of the entire network during training is more stable for high compression schemes. Indeed, by quantizing only a random fraction of the network at each forward, most the weights are updated with unbiased gradients. Interestingly, we show that our method can employ a simpler quantization scheme during the training. This is particularly useful for quantizers with trainable parameters, such as Product Quantizer (PQ), for which our quantization proxy is not parametrized. Our approach simply applies a quantization noise, called Quant-Noise, to a random subset of the weights, see Figure 6-1. We observe that this makes a network resilient to various types of discretization methods: it significantly improves the accuracy associated with (a) low precision representation of weights like int8; and (b) state-of-the-art iPQ. Further, we demonstrate that Quant-Noise can be applied to existing trained networks as a post-processing step, to improve the performance network after quantization. In summary,

• We introduce the Quant-Noise technique to learn networks that are more resilient to a variety of quantization methods such as int4, int8, and iPQ;

• Adding Quant-Noise to iPQ leads to state-of-the-art trade-offs between accuracy and model size. For instance, for natural language processing (NLP), we reach 82.5% accuracy on MNLI by compressing RoBERTa to 14 MB. Similarly for computer vision, we report 80.0% top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3 MB;

• By combining iPQ and int8 to quantize weights and activations for networks trained with Quant-Noise, we obtain extreme compression with fixed-precision computation and achieve 79.8% top-1 accuracy on ImageNet and 21.1 perplexity on WikiText-103.

Related Work

Here, we review the works closest to ours and refer the reader to the Related Work section 2.3 for a more extensive discussion.

Model compression.

Many compression methods focus on efficient parameterization, via weight pruning [START_REF] Lecun | Optimal brain damage[END_REF]Li et al., 2016;Huang et al., 2018a;[START_REF] Mittal | Recovering from random pruning: On the plasticity of deep convolutional neural networks[END_REF], weight sharing [START_REF] Dehghani | Universal transformers[END_REF][START_REF] Turc | Well-read students learn better: The impact of student initialization on knowledge distillation[END_REF][START_REF] Lan | ALBERT: A lite BERT for self-supervised learning of language representations[END_REF] or with dedicated architectures [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF]Zhang et al., 2017b;[START_REF] Howard | Searching for mobilenetv3[END_REF]. Weight pruning is implemented during training (Louizos et al., 2017b) or as a fine-tuning post-processing step [START_REF] Han | Learning both weights and connections for efficient neural network[END_REF](Han et al., , 2016b)). Many pruning methods are unstructured, i.e., remove individual weights [START_REF] Lecun | Optimal brain damage[END_REF][START_REF] Molchanov | Variational dropout sparsifies deep neural networks[END_REF]. On the other hand, structured pruning methods follow the structure of the weights to reduce both the memory footprint and the inference time of a model (Li et al., 2016;[START_REF] Luo | Thinet: A filter level pruning method for deep neural network compression[END_REF][START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF]. We refer the reader to [START_REF] Liu | Rethinking the value of network pruning[END_REF] for a review of different pruning strategies. Other authors have worked on lightweight architectures, by modifying existing models (Zhang et al., 2018a;Wu et al., 2019b;Sukhbaatar et al., 2019a) or developing new networks, such as MobileNet [START_REF] Howard | Searching for mobilenetv3[END_REF], ShuffleNet (Zhang et al., 2017b), and EfficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF]. Finally, knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] has been applied to sentence representation [START_REF] Turc | Well-read students learn better: The impact of student initialization on knowledge distillation[END_REF][START_REF] Sanh | DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[END_REF][START_REF] Zhao | Extreme language model compression with optimal subwords and shared projections[END_REF][START_REF] Jiao | Tinybert: Distilling BERT for natural language understanding[END_REF] and to reduce the size of a BERT model [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF].

Quantization. There are extensive studies of scalar quantization to train networks with low-precision weights and activations [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF][START_REF] Courbariaux | Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1[END_REF][START_REF] Rastegari | Xnornet: Imagenet classification using binary convolutional neural networks[END_REF][START_REF] Mark | Training wide residual networks for deployment using a single bit for each weight[END_REF]. These methods benefit from specialized hardware to also improve the runtime during inference [START_REF] Vanhoucke | Improving the speed of neural networks on cpus[END_REF]. Other quantization methods such as Vector Quantization (VQ) and PQ [START_REF] Jegou | Product quantization for nearest neighbor search[END_REF] quantize blocks of weights simultaneously to achieve higher compression rate [START_REF] Gong | Compressing deep convolutional networks using vector quantization[END_REF][START_REF] Joulin | Fasttext.zip: Compressing text classification models[END_REF][START_REF] Miguel | Model compression as constrained optimization, with application to neural nets[END_REF]. Closer to our work, several works have focused at simultaneously training and quantizing a network [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF][START_REF] Krishnamoorthi | Quantizing deep convolutional networks for efficient inference: A whitepaper[END_REF][START_REF] Gupta | Deep learning with limited numerical precision[END_REF][START_REF] Dong | Stochastic quantization for learning accurate low-bit deep neural networks[END_REF]. [START_REF] Gupta | Deep learning with limited numerical precision[END_REF] assigns weights to a quantized bin stochastically which is specific to scalar quantization, but permits training with fixed point arithmetic. Finally, our method can be interpreted as a form of Bayesian compression (Louizos et al., 2017b), using the Bayesian interpretation of Dropout [START_REF] Gal | Dropout as a bayesian approximation: Representing model uncertainty in deep learning[END_REF]. However, we select our noise to match the weight transformation of a quantization method without restricting it to a scale mixture prior.

Quantizing Neural Networks

In this section, we present the principles of quantization, several standard quantization methods, and describe how to combine scalar and product quantization. For clarity, we focus on the case of a fixed real matrix 𝑊 ∈ R 𝑛×𝑝 . We suppose that this matrix is split into 𝑚 × 𝑞 blocks 𝑏 𝑘𝑙 as follows:

𝑊 = ⎛ ⎜ ⎜ ⎜ ⎝ 𝑏 11 • • • 𝑏 1𝑞 . . . . . . . . . 𝑏 𝑚1 • • • 𝑏 𝑚𝑞 ⎞ ⎟ ⎟ ⎟ ⎠ , (6.1)
where the nature of these blocks is determined by the quantization method We distinguish scalar quantization, such as int8, where each block 𝑏 𝑘𝑙 consists of a single weight, from vector quantization, where several weights are quantized jointly.

Fixed-point Scalar Quantization

Fixed-point scalar quantization methods replace floating-point number representations by low-precision fixed-point representations. They simultaneously reduce a model's memory footprint and accelerate inference by using fixed-point arithmetic on supporting hardware.

Fixed-point scalar quantization operates on blocks that represent a single weight, i.e., 𝑏 𝑘𝑙 = 𝑊 𝑘𝑙 . Floating-point weights are replaced by 𝑁 bit fixed-point numbers [START_REF] Gupta | Deep learning with limited numerical precision[END_REF], with the extreme case of binarization where 𝑁 = 1 [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF]. More precisely, the weights are rounded to one of 2 𝑁 possible codewords. These codewords correspond to bins evenly spaced by a scale factor 𝑠 and shifted by a bias 𝑧. Each weight 𝑊 𝑘𝑙 is mapped to its nearest codeword 𝑐 by successively quantizing with 𝑧 ↦ → round(𝑊 𝑘𝑙 /𝑠 + 𝑧) and dequantizing with:

𝑐 = (round(𝑊 𝑘𝑙 /𝑠 + 𝑧) -𝑧) × 𝑠, (6.2)
where we compute the scale and bias as:

𝑠 = max 𝑊 -min 𝑊 2 𝑁 -1 and 𝑧 = round(min 𝑊/𝑠).
We focus on this uniform rounding scheme instead of other non-uniform schemes [START_REF] Choi | Pact: Parameterized clipping activation for quantized neural networks[END_REF][START_REF] Li | Additive powers-of-two quantization: A nonuniform discretization for neural networks[END_REF], because it enables fixed-point arithmetic with implemen-tations in PyTorch and Tensorflow (see Appendix). The compression rate is ×32/𝑁 . The activations are also rounded to 𝑁 -bit fixed-point numbers. With int8 for instance, this leads to ×2 to ×4 faster inference on dedicated hardware. In this work, we consider both int4 and int8 quantization.

Product Quantization

Several quantization methods work on groups of weights, such as vectors, to benefit from the correlation induced by the structure of the network. In this work, we focus on Product Quantization for its good performance at extreme compression ratio as explained in Chapter 5.

Traditional PQ. In vector quantization methods, the blocks are predefined groups of weights instead of single weights. The codewords are groups of values, and the index matrix 𝐼 maps groups of weights from the matrix 𝑊 to these codewords. In this section, we present the Product Quantization framework as it generalizes both scalar and vector quantization. We consider the case where we apply PQ to the columns of 𝑊 and thus assume that 𝑞 = 𝑝.

Traditional vector quantization techniques split the matrix 𝑊 into its 𝑝 columns and learn a codebook on the resulting 𝑝 vectors. Instead, Product Quantization splits each column into 𝑚 subvectors and learns the same codebook for each of the resulting 𝑚 × 𝑝 subvectors. Each quantized vector is subsequently obtained by assigning its subvectors to the nearest codeword in the codebook. Learning the codebook is traditionally done using 𝑘-means with a fixed number 𝐾 of centroids, typically 𝐾 = 256 to store the index matrix 𝐼 using int8. Thus, the objective function is written as:

‖𝑊 -̂ 𝑊 ‖ 2 2 = ∑ 𝑘,𝑙 ‖𝑏 𝑘𝑙 -𝑐[𝐼 𝑘𝑙 ]‖ 2 2 . (6.3)
PQ shares representations between subvectors, yielding higher compression rates than int8 or int4 (respectively ×4 and ×8 with respect to the non-compressed model).

iPQ. When quantizing a full network rather than a single matrix, extreme compression with PQ induces a quantization drift as reconstruction error accumulates as explained in Chapter 5. Indeed, subsequent layers take as input the output of preceding layers, which are modified by the quantization of the preceding layers. This creates a drift in the network activations, resulting in large losses of performance.

A solution proposed in Chapter 5, called iterative PQ (iPQ), is to quantize layers sequentially from the lowest to the highest, and finetune the upper layers as the lower layers are quantized, under the supervision of the uncompressed (teacher) model. Codewords of each layer are finetuned by averaging the gradients of their assigned elements with gradient steps:

𝑐 ← 𝑐 -𝜂 1 |𝐽 𝑐 | ∑ (𝑘,𝑙)∈𝐽𝑐 𝜕ℒ 𝜕𝑏 𝑘𝑙 , ( 6.4)
where

𝐽 𝑐 = {(𝑘, 𝑙) | 𝑐[𝐼 𝑘𝑙 ] = 𝑐}, ℒ
is the loss function and 𝜂 > 0 is a learning rate. This adapts the upper layers to the drift appearing in their inputs, reducing the impact of the quantization approximation on the overall performance.

Combining Fixed-Point with Product Quantization

Fixed-point quantization and Product Quantization are often regarded as competing choices, but can be advantageously combined. Indeed, PQ/iPQ compresses the network by replacing vectors of weights by their assigned centroids, but these centroids are in floating-point precision. Fixed-point quantization compresses both activations and weights to fixed-point representations. Combining both approaches means that the vectors of weights are mapped to centroids that are compressed to fixed-point representations, along with the activations. This benefits from the extreme compression ratio of iPQ and the finite-precision arithmetics of intN quantization.

More precisely, for a given matrix, we store the int8 representation of the 𝐾 centroids of dimension 𝑑 along with the log 2 𝐾 representations of the centroid assignments of the 𝑚 × 𝑝 subvectors. The int8 representation of the centroids is obtained with Eq. (6.2). The overall storage of the matrix and activations during a forward pass with batch size 1 (recalling that the input dimension is n) writes

𝑀 = 8 × 𝐾𝑑 + log 2 𝐾 × 𝑚𝑝 + 8 × 𝑛 bits. (6.5)
In particular, when 𝐾 = 256, the centroid assignments are also stored in int8, which means that every value required for a forward pass is stored in an int8 format. We divide by 4 the float32 overhead of storing the centroids, although the storage requirement associated with the centroids is small compared to the cost of indexing the subvectors for standard networks. In contrast to iPQ alone where we only quantize the weights, we also quantize the activations using int8.

Method

Deep networks are not exposed to the noise caused by the quantization drift during training, leading to suboptimal performance. A solution to make the network robust to quantization is to introduce it during training. Quantization Aware Training (QAT) [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] exposes the network during training by quantizing weights during the forward pass. This transformation is not differentiable and gradients are approximated with a straight through estimator (STE) [START_REF] Bengio | Estimating or propagating gradients through stochastic neurons for conditional computation[END_REF][START_REF] Courbariaux | Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1[END_REF]. STE introduces a bias in the gradients that depends on level of quantization of the weights, and thus, the compression ratio. In this section, we propose a simple modification to control this induced bias with a stochastic amelioration of QAT, called Quant-Noise. The idea is to quantize a randomly selected fraction of the weights instead of the full network as in QAT, leaving some unbiased gradients flow through unquantized weights. Our general formulation can simulate the effect of both quantization and of pruning during training.

Training Networks with Quantization Noise

We consider the case of a real matrix 𝑊 as in Section 6.3. During the training of a network, our proposed Quant-Noise method works as follows: first, we compute blocks 𝑏 𝑘𝑙 related to a target quantization method. Then, during each forward pass, we randomly select a subset of these blocks and apply some distortion to them, and we compute gradients for all the weights, using STE for the distorted weights.

More formally, given a set of tuples of indices 𝐽 ⊂ {(𝑘, 𝑙)} for 1 ≤ 𝑘 ≤ 𝑚, 1 ≤ 𝑙 ≤ 𝑞 and a distortion or noise function 𝜙 acting on a block, we define an operator 𝜓(• | 𝐽) such that, for each block 𝑏 𝑘𝑙 , we apply the following transformation:

𝜓(𝑏 𝑘𝑙 | 𝐽) = ⎧ ⎪ ⎨ ⎪ ⎩ 𝜙(𝑏 𝑘𝑙 ) if (𝑘, 𝑙) ∈ 𝐽, 𝑏 𝑘𝑙 otherwise.
(6.6)

The noise function 𝜙 simulates the change in the weights produced by the target quantization method (see Section 6.4.2 for details). We replace the matrix 𝑊 by the noisy matrix 𝑊 noise during the forward pass to compute a noisy output 𝑦 noise as

𝑊 noise = (𝜓(𝑏 𝑘𝑙 | 𝐽)) 𝑘𝑙 and 𝑦 noise = 𝑥𝑊 noise (6.7)
where 𝑥 is an input vector. During the backward pass, we apply STE, which amounts to replacing the distorted weights 𝑊 noise by their non-distorted counterparts. Note that our approach is equivalent to QAT when 𝐽 contains all the tuples of indices. However, an advantage of Quant-Noise over QAT is that unbiased gradients continue to flow via blocks unaffected by the noise. As these blocks are randomly selected for each forward, we guarantee that each weight regularly sees gradients that are not affected by the nature of the function 𝜙. As a side effect, our quantization noise regularizes the network in a similar way as DropConnect [START_REF] Wan | Regularization of neural networks using DropConnect[END_REF] or LayerDrop [START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF].

Composing quantization noises. As noise operators are compositionally commutative, we can make a network robust to a combination of quantization methods by composing their noise operators:

𝜓(𝑏 𝑘𝑙 | 𝐽) = 𝜓 1 ∘ 𝜓 2 (𝑏 𝑘𝑙 | 𝐽). (6.8)
This property is particularly useful to combine quantization with pruning operators during training, as well as combining scalar quantization with product quantization.

Adding Noise to Specific Quantization Methods

In this section, we propose several implementations of the noise function 𝜙 for the quantization methods described in Section 6.3.

Fixed-point scalar quantization. In intN quantization, the blocks are atomic and weights are rounded to their nearest neighbor in the codebook. The function 𝜙 replaces weight 𝑊 𝑘𝑙 with the output of the rounding function defined in Eq. ( 6.2)

𝜙 intN (𝑤) = (round(𝑤/𝑠 + 𝑧) -𝑧) × 𝑠, ( 6.9)
where 𝑠 and 𝑧 are updated during training. In particular, the application of Quant-Noise to int8 scalar quantization is a stochastic amelioration of QAT.

Product quantization. As opposed to intN, codebooks in PQ require a clustering step based on weight values. During training, we learn codewords online and use the resulting centroids to implement the quantization noise. More precisely, the noise Quant-Noise on language modeling and image classification. For language modeling, we train a Transformer on the Wikitext-103 benchmark and report perplexity (PPL) on test.

For image classification, we train a EfficientNet-B3 on the ImageNet-1k benchmark and report top-1 accuracy on validation and use our re-implementation of EfficientNet-B3. The original implementation of [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] achieves an uncompressed Top-1 accuracy of 81.9%. For both settings, we report model size in megabyte (MB) and the compression ratio compared to the original model.

function 𝜙 PQ assigns a selected block 𝑏 to its nearest codeword in 𝒞:

𝜙 PQ (𝑣) = argmin 𝑐∈𝒞 ‖𝑏 -𝑐‖ 2 2 . (6.10)
Updating the codebooks online works well. However, empirically, running 𝑘-means once per epoch is faster and does not noticeably modify the resulting accuracy. Note that computing the exact noise function for PQ is computationally demanding. We propose a simpler and faster alternative approximation 𝜙 proxy to the operational transformation of PQ and iPQ. The noise function simply zeroes out the subvectors of the selected blocks, i.e., 𝜙 proxy (𝑣) = 0. As a side note, we considered other alternatives, for instance one where the subvectors are mapped to the mean subvector. In practice, we found that these approximations lead to similar performance, see Section C.2. This proxy noise function is a form of Structured Dropout and encourages correlations between the subvectors. This correlation is beneficial to the subsequent clustering involved in PQ/iPQ. Adding pruning to the quantization noise. The specific form of quantization noise can be adjusted to incorporate additional noise specific to pruning. We simply combine the noise operators of quantization and pruning by composing them following Eq. (6.8). We consider the pruning noise function of [START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF] where they randomly drop predefined structures during training. In particular, we focus on LayerDrop, where the structures are the residual blocks of highway-like layers [START_REF] Kumar Srivastava | Highway networks[END_REF], as most modern architectures, such as ResNet or Transformer, are composed of this structure. More precisely, the corresponding noise operator over residual blocks 𝑣 is 𝜙 LayerDrop (𝑣) = 0. For pruning, we do not use STE to backpropagate the gradient of pruned weights, as dropping them entirely during training has the benefit of speeding convergence [START_REF] Huang | Deep networks with stochastic depth[END_REF]. Once a model is trained with LayerDrop, the number of layers kept at inference can be adapted to match computation budget or time constraint.

Experiments

We demonstrate the impact of Quant-Noise on the performance of several quantization schemes in a variety of settings (see Appendix -Sec. C.3).

Improving Compression with Quant-Noise

Quant-Noise is a regularization method that makes networks more robust to the target quantization scheme or combination of quantization schemes during training. We show the impact of Quant-Noise in Table 1 for a variety of quantization methods. We experiment in 2 different settings: a Transformer network trained for language modeling on WikiText-103 and a EfficientNet-B3 convolutional network trained for image classification on ImageNet-1k. Our quantization noise framework is general and flexible -Quant-Noise improves the performance of quantized models for every quantization scheme in both experimental settings. Importantly, Quant-Noise only changes model training by adding a regularization noise similar to dropout, with no impact on convergence and very limited impact on training speed (< 5% slower). This comparison of different quantization schemes shows that Quant-Noise works particularly well with high performance quantization methods, like iPQ, where QAT tends to degrade the performances, even compared to quantizing as a post-processing step. In subsequent experiments in this section, we focus on applications with iPQ because it offers the best trade-off between model performance and compression, and has little negative impact on FLOPS.

Fixed-Point Product Quantization. Combining iPQ and int8 as described in Section 6.3.3 allows us to take advantage of the high compression rate of iPQ with a fixed-point representation of both centroids and activations. As shown in Table 6.1, this combination incurs little loss in accuracy with respect to iPQ + Quant-Noise. Most of the memory footprint of iPQ comes from indexing and not storing centroids, so the compression ratios are comparable.

Complementarity with Weight Pruning and Sharing.

We analyze how Quant-Noise is compatible and complementary with pruning ("+Prune") and weight sharing ("+Share"), see Appendix for details on weight sharing. We report results for Language modeling on WikiText-103, pre-trained sentence representations on MNLI and object classification on ImageNet-1k in Table 6.2. The conclusions are remarkably consistent across tasks and benchmarks: Quant-Noise gives a large improvement over strong iPQ baselines. Combining it with sharing and pruning offers additional interesting operating points of performance vs size.

Comparison with the State of the Art

We now compare our approach on the same tasks against the state of the art. We compare iPQ + Quant-Noise with 6 methods of network compression for Language modeling, 8 state-of-the-art methods for Text classification, and 8 recent methods evaluate image classification on ImageNet with compressed models. These comparisons demonstrate that Quant-Noise leads to extreme compression rates at a reasonable cost in accuracy. We apply our best quantization setup on competitive models and reduce their memory footprint by ×20 -94 when combining with weight sharing and pruning, offering extreme compression for good performance.

Natural Language Processing. In Figure 6-2, we examine the trade-off between performance and model size. Our quantized RoBERTa offers a competitive trade-off between size and performance with memory reduction methods dedicated to BERT, like TinyBERT, MobileBERT, or AdaBERT.

Image Classification. We compress EfficientNet-B3 from 46.7Mb to 3.3Mb (×14 compression) while maintaining high top-1 accuracy (78.5% versus 80% for the original model). As shown in Figure 6-2, our quantized EfficientNet-B3 is smaller and more accurate than architectures dedicated to optimize on-device performance with limited size like MobileNet or ShuffleNet. We further evaluate the beneficial effect of Quant-Noise on ResNet-50 to compare directly with the results of Chapter 5.

Incorporating pruning noise into quantization is also beneficial. For example, with pruning iPQ+Quant-Noise reduces size by ×25 with only a drop of 2.4 PPL in language modeling. Further, pruning reduces FLOPS by the same ratio as its compression factor, in our case, ×2. By adding sharing with pruning, in language modeling, we achieve an extreme compression ratio of ×94 with a drop of 6.4 PPL with FLOPS reduction from pruning entire shared chunks of layers. For comparison, our 10 MB model has the same performance as the 570 MB Transformer-XL base.

Finetuning with Quant-Noise

We explore taking existing models and post-processing with Quant-Noise instead of training from scratch. For language modeling, we train for 10 additional epochs.

For RoBERTa, we train for 25k additional updates. Finetuning with Quant-Noise incorporates the benefits and almost matches training from scratch (Table 6.3). In language modeling, there is only a 0.2 PPL difference.

Conclusion

We show that quantizing a random subset of weights during training maintains performance in the high quantization regime. We validate that Quant-Noise works with a variety of different quantization schemes on several applications in text and vision.

Our method can be applied to a combination of iPQ and int8 to benefit from extreme compression ratio and fixed-point arithmetic. Finally, we show that Quant-Noise can be used as a post-processing step to prepare already trained networks for subsequent quantization, to improve the performance of the compressed model.

Chapter 7 Compressing Faces for Ultra-Low Bandwidth Video Chat

To unlock video chat for hundreds of millions of people hindered by poor connectivity or unaffordable data costs, we propose to authentically reconstruct faces on the receiver's device using facial landmarks extracted at the sender's side and transmitted over the network. In this context, we discuss and evaluate the benefits and disadvantages of several deep learning approaches. In particular, we explore quality and bandwidth trade-offs for approaches based on static landmarks, dynamic landmarks or segmentation maps. We design a mobile-compatible architecture based on the first order animation model of [START_REF] Siarohin | First order motion model for image animation[END_REF]. In addition, we leverage SPADE blocks [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF] to refine results in important areas such as the eyes and lips. We compress the networks down to about 3 MB, allowing models to run in real time on iPhone 8 (CPU). This approach enables video calling at a few kbits per second, an order of magnitude lower than currently available alternatives.

Introduction

For many smartphone users around the world, video-calling remains unavailable or unaffordable. These users are driven out of this fundamental connectivity experience by the prohibitive cost of data plans or because they depend on outdated technologies and infrastructures. For instance, networks might suffer from congestion, poor coverage, power fluctuations and datarate limits -2G networks allow for a maximum of 30 kbits/s. However, with current technologies, an acceptable video-call quality requires at least a stable 200 kbits/s connection.

Meanwhile, the research in generative models has now come to a point where the Figure 7-1: We propose to authentically reconstruct faces in real-time on mobile devices using a stream of compressed facial landmarks extracted from driving or target frames. The identity of the sender is transmitted one-shot to the receiver at the beginning of the call through a reference or source frame. This approach is compatible with end-to-end encryption (E2EE).

quality of synthetic faces are sometimes indistinguishable from real videos [START_REF] Dolhansky | The deepfake detection challenge (dfdc) dataset[END_REF]. To name a few, we may cite Deep video portraits [START_REF] Kim | Deep video portraits[END_REF], X2Face [START_REF] Wiles | X2face: A network for controlling face generation using images, audio, and pose codes[END_REF], FSGAN [START_REF] Nirkin | FSGAN: Subject agnostic face swapping and reenactment[END_REF], Neural Talking Heads [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF], the Bilayer model [START_REF] Zakharov | Fast bi-layer neural synthesis of one-shot realistic head avatars[END_REF] and the First Order Model [START_REF] Siarohin | First order motion model for image animation[END_REF]. This unprecedented performance can now be exploited to the benefit of higher quality video calls. However, there remain important challenges to address before generative models can offer ultra-low data-rate video-calling. In particular, to unlock duplex video-calling for users with last-mile connectivity issues or limited data plans, the models need to be light enough to run on mobile handsets. Generating the user's face in real time on device is compatible with end-to-end encryption. In addition, to deliver a more seamless and authentic experience, the models should adapt to the current appearance of the user without additional training. In this work, we focus on identifying the best generative strategy compatible with real-time inference on device. We discuss the following approaches:

• The Neural Talking Heads model [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF], which requires sending a stream of landmarks in addition to an initial face embedding.

• The Bilayer model [START_REF] Zakharov | Fast bi-layer neural synthesis of one-shot realistic head avatars[END_REF], where the face is reconstructed from a stream of landmarks and a reference frame sent once.

• The SegFace model, a novel architecture based on SPADE [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF], adapted to face animation, which requires sending an initial face embedding and a stream of semantic segmentation maps.

• The First Order Model (FOM) [START_REF] Siarohin | First order motion model for image animation[END_REF], which requires sending ten landmarks, their associated motion matrices, and one reference frame.

Analyzing the FOM in depth, we observe that only sending the landmarks compressed with Huffman coding (no motion matrices) achieves sufficient quality and leads to an outstanding data-rate reduction. Compared to other approaches, this model allows for good identity and background preservation. In summary, our contributions are the following:

• We provide a comparative analysis of leading generative approaches for the specific use-case of enabling ultra-low data-rate video calling.

• We develop a strong baseline leveraging the SPADE architecture and the segmentation maps.

• We propose a warping based approach leveraging SPADE blocks to refine important face attributes such as eyes and lips.

• While previous approaches were tested on specialized hardware (servers, mobile GPU), we provide first real-time results on mobile CPU.

Related Work

Face Compression Before Deep Learning

The idea of face-specific video compression is not novel and appeared with classical computer vision tools, for instance morphings using Delaunay triangulations, Eigenfaces, or 3D models. The first reference we found on the topic is the work of [START_REF] Lopez | Head pose computation for very low bitrate video coding[END_REF] that proposes to encode only pose parameters of a 3D head model, which is projected to reproduce a video sequence.

Previous work [START_REF] Koufakis | Very low bit rate face video compression using linear combination of 2D face views and principal components analysis[END_REF]) use PCA to model the current frame as a linear combination of 3 basis frames sent prior to the call. The authors rely on known control points on the face boundaries and landmarks. The principal drawback of the approach is the presence of triangulation artifacts, even when a large number of control points is used. The achieved bandwidth is 1500 bits/frame. Similar usage of Eigenspaces are suggested in [START_REF] Tuceryan | Model based faced coding and decoding using feature detection and eigenface coding[END_REF][START_REF] Torres | A proposal for high compression of faces in video sequences using adaptive eigenspaces[END_REF][START_REF] Söderström | Very low bitrate facial video coding: based on principal component analysis[END_REF]. Among these proposals using eigenspaces, one claims an extremely low bit-rates achievement of 100 bits/s [START_REF] Son | Ultra low bit-rate video communication[END_REF]. However the proposed solution is hard to scale, as it requires storing personal galleries of face images to reconstruct videos at the receiver side.

Deep Compression

The emergence of Generative Adversarial Networks (GANs) stimulated the application of deep learning to video compression. Super-resolution has been an active field of research leveraging GANs for image and video compression. There have been a number of research works tackling this problem [START_REF] Chen | Fsrnet: End-to-end learning face super-resolution with facial priors[END_REF][START_REF] Ustinova | Deep multi-frame face hallucination for face identification[END_REF][START_REF] Bulat | Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans[END_REF]. However, for compressing faces, these reconstructions methods are limited to restoring personal traits from low level images and only work well for limited upscaling factors (around 2× in resolution). The power of GANs for lossy image compression started to be demonstrated in the Generative compression work of Santurkar et al. (2018a), using an auto-encoder combined with adversarial training. The state-of-the-art has since improved with the Extreme Learned Image Compression work of [START_REF] Agustsson | Generative adversarial networks for extreme learned image compression[END_REF], thanks to a multi-scale architecture and the usage of semantic segmentation information, among other tricks used by the authors. The work of [START_REF] Liu | Deep learning-based video coding: A review and a case study[END_REF] surveys deep learningbased approaches for general purpose video compression. Among them, Learned Video Compression [START_REF] Rippel | Learned video compression[END_REF] demonstrates for the first time the superior capacity of an end-to-end machine learning approach over standard codecs. By focusing on faces only, we can lower the bandwidth, improve the quality and compress models compared to using more generic methods. Therefore, we review next deep learning approaches and their adequacy to video chat compression.

Deep Talking Head Approaches

3D based approaches produce realistic avatars which can be animated in real-time [START_REF] Cao | Real-time facial animation with image-based dynamic avatars[END_REF]. However, such methods require to capture a set of images of the user (a few dozens) to build their personal face model. PAGAN [START_REF] Nagano | paGAN: real-time avatars using dynamic textures[END_REF] generates key face expression textures that can be deformed and blended in realtime on mobile from a single frame. However, the reconstruction of certain features, notably the hair, is still problematic in 3D model-based approaches. Deep video portraits [START_REF] Kim | Deep video portraits[END_REF] is handling this issue using a rendering-to-video trans-lation network, but the approach needs about a thousand images per subject for training. Stimulated by advancements in face swapping pipelines [START_REF] Korshunova | Fast face-swap using convolutional neural networks[END_REF][START_REF] Wiles | X2face: A network for controlling face generation using images, audio, and pose codes[END_REF], a number of deep generative re-enactment approaches arose. Contrary to warping based re-enactment [START_REF] Averbuch-Elor | Bringing portraits to life[END_REF], learning faces reconstructions enables extra robustness in presence of large head angles. The Face Swapping GAN [START_REF] Nirkin | FSGAN: Subject agnostic face swapping and reenactment[END_REF] relies on several steps: landmarks extraction, segmentation, interpolation and inpainting. This complex pipeline may result in robustness issues and limited bandwidth gain due to the need of sending both compressed segmentations and landmarks. Similarly, the vid2vid approach ( (Wang et al., 2018b), (Wang et al., 2019b)) requires sending a "sketch" (edge map) for each frame in order to re-enact a face, which has a relatively high bandwidth cost.

Generative Models

In this section we describe in-depth several recent face animation algorithms that we have implemented and studied. We share our understanding of these works and present our two model contributions, namely SegFace and Hybrid Motion-SPADE. An overview of these different models appears in Figure 7-2. For this self-reenactment task, unless mentioned otherwise, the goal of all these approaches is to generate a frame based on (i) one fixed reference or source frame and (ii) position information (e.g. landmarks) from a stream of driving or target frames (see Figure 7-1). Implementation details are located in the original paper of [START_REF] Oquab | Low bandwidth video-chat compression using deep generative models[END_REF].

Talking Heads (NTH) and Bilayer Model

The "Talking Heads" work of [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF] learns to synthesize videos of people from facial landmarks given one reference image. It follows an encoder-decoder architecture with a style transfer component. A set of style parameters is computed for the set of reference images. Then, facial landmarks are plotted as an image and processed by an encoder network. The resulting code is decoded with style transfer, using Adaptive Instance Normalization [START_REF] Huang | Arbitrary style transfer in real-time with adaptive instance normalization[END_REF] layers, adjusting the mean and standard deviation of each feature map with the style parameters.

The networks are trained end-to-end with adversarial and perceptual losses on a dataset of videos. The best results are achieved by performing a fine-tuning training phase on the generator to match the reference frames as precisely as possible. This fine-tuning phase requires several minutes on a modern server GPU. Without fine-Figure 7-2: Scheme of principle for the different deep generative approaches discussed in this study. In particular, we detail two novel architectures, SegFace and Hybrid Motion-SPADE (right) and compare them to existing NTH [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF] and FOM (Siarohin et al., 2019) models (left). For all models, we assume the generation is performed by the encoder-decoder pair on the receiver device, while the emitter sends a reference frame (or several) at the beginning of inference, and streams a series of landmarks or segmentation maps.

tuning, the identity is not preserved as well in the generated frames. In practice, a few hundreds of frames would have to be sent at the beginning of the call. This work was further improved in the Bilayer Synthesis approach [START_REF] Zakharov | Fast bi-layer neural synthesis of one-shot realistic head avatars[END_REF], where the fine-tuning step is not required anymore, and leads to visually appealing and sharp results. In our observations (see Figure 7-3), the identity preservation suffers from a stronger uncanny valley effect. In terms of bandwidth, the NTH and Bilayer approaches require sending 68 compressed landmarks.

First Order Model for Image Animation (FOM)

The "First Order Model" approach of [START_REF] Siarohin | First order motion model for image animation[END_REF] deforms a reference source frame to follow the motion of a driving video. While this method works on various types of videos (Tai-chi, cartoons), we focus here on the face animation application. FOM follows an encoder-decoder architecture with a motion transfer component. Fist, a landmark extractor is learned using an equivariant loss, without explicit labels. Then, two sets of ten learned landmarks are computed for the source and driving frames, and a dense motion network uses the landmarks and the source frame to produce a dense motion field and an occlusion map. In parallel, the encoder encodes the source frame. Next, the resulting feature map is warped using the dense motion field (through a grid-sample operation [START_REF] Jaderberg | Spatial transformer networks[END_REF]), then multiplied with the occlusion map. Finally, the decoder generates an image from the warped map.

The networks are trained end-to-end on video frames, using perceptual losses, and are then optionally fine-tuned with an adversarial discriminator. The self-supervised landmarks do not necessarily match precise locations of the face. Instead, they correspond to point coordinates that are optimized to achieve the best deformation of the source frame. [START_REF] Siarohin | First order motion model for image animation[END_REF] describes how to improve motion approximation in landmark areas by estimating Jacobian matrices to model motion in their neighborhood. In our observations (see Table 7.3), this approach preserves identities better than NTH and is at least on par with the follow-up Bilayer synthesis approach. Next, we study variants of this approach.

Variants First, our implementation does not use the Jacobian component, as we do not observe a strong influence on the results. We refer to the resulting model as "Motion Net (MN-10)" as it no longer uses first order approximation and employs a set of ten landmarks. Second, we explore using off-the-shelf facial landmarks extraction to complement the unsupervised landmarks. In this case, we only stream 20 or 68 compressed landmarks. Third, we explore a combined strategy employing both 10 self-supervised landmarks and 20 supervised ones, that we note MN-10+20. We will introduce a fourth variant in 7.3.4, after detailing our SegFace approach below.

SegFace

This novel approach builds upon [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF]. Unlike MaskGAN (Lee et al., 2020), we propose to use a face descriptor computed on a source frame, and decode it by conditioning on face segmentation maps from a driving frame. It follows an encoder-decoder architecture: a face descriptor is computed on a source frame and given to a decoder network, that applies SPADE normalization blocks at each layer using the face segmentation maps of the driving frame, ensuring all parts of the face are correctly placed. The decoder network is trained using VGGFace2 face embeddings [START_REF] Cao | Vggface2: A dataset for recognising faces across pose and age[END_REF], and segmentation maps from Yu et al. (2018a) as inputs. Its objective during training is to reconstruct the same source frame. The optimization is done using losses from [START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF], and the face perceptual loss from [START_REF] Gafni | Live face de-identification in video[END_REF]. This method operates on independent frames, and thus allows us to use high-resolution training data, leading to high quality results. Training is achieved using CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF] and Flickr-Faces-HQ datasets.

Bandwidth The model requires a segmentation map labeled for 15 categories (eyes, hairs, ears etc.). Sending compressed segmentation maps would require 18/25 kbits/s at resolutions 48×/64×, knowing that there is a trade-off between the resolution of the transmitted segmentation maps and the quality of the generated faces. We do not build on this method further for low-bandwidth video-chat because the cost of running a face parser inference step and the bandwidth requirements are too high. The SegFace implementation, however, allows us to observe that the generated images respect the segmentation map labels almost perfectly, consistently with the conclusions of Lee et al. (2020). We will build on this property in the next subsection with our Hybrid Motion-SPADE approach.

Hybrid Motion-SPADE Model

Important quality criteria for compressed video-chat include a good synchronization between the lips and the speech, and a good rendering of the eyes and eyebrows; therefore, it is crucial to generate these facial parts precisely. We propose an improvement over the FOM-based Motion Net method, by adding SPADE normalization layers in the upsampling blocks of the decoder network (in the last step of the FOM approach). We draw polygons for the eyes, eyebrows, lips and inner mouth using 60 extracted face landmarks, and use these as semantic maps for SPADE.

The dense motion network receives (i) a downsampled reference frame with (ii) the positions of 𝑁 landmarks for that frame, and (iii) the positions of the same landmarks for a driving frame. It outputs a motion field 𝑀 and an occlusion map 𝑂. The encoder network outputs a feature map 𝐹 𝑠 . The decoder warps 𝐹 𝑠 with the result of the dense motion network 𝑀 and multiplies it element-wise with the occlusion map 𝑂, to obtain 𝐹 𝑤 . Then, 𝐹 𝑤 is processed by a stack of five residual blocks and three upsampling blocks that apply the SPADE normalization using a set of 60 landmarks.

Bandwidth

The necessary segmentation maps are obtained by plotting the polygons of the facial landmarks extracted using a landmark detector (see Figure 7-2), rather than running a face segmentation network. Moreover, landmark coordinates are inexpensive to transmit, while rasterized segmentation maps are more difficult to compress, especially at higher resolutions. In terms of bandwidth, this approach requires sending 𝑁 + 60 compressed landmarks. We experiment with 𝑁 = 10, 20, 30.

Compression

In this section, we explain different strategies to make architectures -and in particular our novel hybrid Motion-SPADE -compatible with low-bandwidth video calls on mobile. We first detail the architectures and then the compression aspects for the models and the bandwidth. Results are displayed in Table 7.1.

Mobile Architectures

Base blocks We rely on the open-source FbNet family of architectures [START_REF] Wan | FBNetV2: Differentiable neural architecture search for spatial and channel dimensions[END_REF][START_REF] Dai | Fbnetv3: Joint architecture-recipe search using neural acquisition function[END_REF] to design mobile-capable models for our Motion Net and Motion-SPADE approaches. These networks typically build on blocks combining 1 × 1 pointwise and 3 × 3 depthwise convolutions (Sandler et al., 2018a) that require less floating-point operations than traditional 3 × 3 convolutions in residual blocks.

Mobile SPADE normalization blocks

When applicable, we perform a SPADE normalization after the last 1 × 1 pointwise convolution, with kernel sizes of 1 × 1, and 32 hidden channels. We have found these parameters to provide a good tradeoff between speed and quality while preserving the fidelity of the SPADE approach.

Landmark Stream Compression

We compress the landmarks with Huffman encoding [START_REF] Huffman | A method for the construction of minimum-redundancy codes[END_REF]. In this approach, the landmark displacements are first binarized into 32 bins plus one bit sign, and we encode the bin index with Huffman coding. This compression leads to an average rate of 130 bits/frame for 20 landmarks, hence 3 kbits/s at 25 FPS (see Table 7.1 for details). For reference, bandwidth requirements for audio are around 10 kbits/s, while the AV1 video codec (not widely hardware-supported to date) aims at 30 kbits/s [START_REF] Citron | four new google duo features to help you stay connected[END_REF]. Therefore, we did not explore other variants such as Arithmetic Coding [START_REF] Rissanen | Arithmetic coding[END_REF] since the audio part takes most of the bandwidth of a call with the proposed approach.

Model Quantization

We rely on int8 post-training quantization. As detailed in Section 2.3.5 of the Related Work, this technique consists in uniformly quantizing both weights and activations over 8 bits, reducing the model size by a factor 4. Moreover, int8 models traditionally benefit from a ×2 or ×3 speed-up compared to their fp32 counterparts for both server and mobile CPUs. The scale and zero-point1 are calibrated after training using a few batches of training data. When not properly calibrated, we found that the decoder generates an image with a small amount of noise resulting in a loss of visual quality.

Experiments

Quantitative Evaluation

We evaluate the models using the perceptual LPIPS (Zhang et al., 2018b) and multiscale LPIPS-like metrics employed in [START_REF] Siarohin | First order motion model for image animation[END_REF], that we name msVGG. Second, as argued in Chen et al. (2020b), the cosine similarity CSIM computed between features of the pre-trained face embedding network ArcFace [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF] is one of the most effective metric to assess quality of talking heads models, we therefore report it. Finally, we quantify facial landmarks mismatch by running a landmark detector on the true and generated videos and computing the Mean Square Error between each pair of landmarks. This metric is classically referred to as the Normalized Mean Error (NME) of head pose [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)[END_REF]. All the generative approaches considered in this work are trained using different alignments and closeups (see Figure 7-3), so we compute our metrics using the optimal modified videos for each method as ground truth. Ablation studies are displayed in Appendix D.1.1 and the quantitative evaluation of the models is displayed in Table D.2.

Figure 7-3: Comparison of different results using Seg2Face (48 × 48), NTH, Bilayer, and FOM adv. The model generates the face using the fixed source frame and the facial information (such as keypoints) of the driving frame. We pasted ground truth backgrounds to have a fair evaluation. The last column showcases the results using our Mobile Motion-SPADE that runs at 18 FPS on an iPhone 8, whereas the other models run on server, have at least 10× more parameters and are not necessarily compatible with low-bandwidth video calling. Note that the alignment procedure may differ between the models, hence the head is not centered the same on the generated faces (last 5 columns).

Qualitative Evaluation and Human Study

Figure 7-3 compares results obtained using SegFace, Bilayer, NTH and FOM with adversarial finetuning. We observe skin tones differences between targets and SegFace results and distortions of personal traits. The Bilayer, NTH and FOM models are qualitatively better. In the last column, we observe a side by side comparison with a Mobile Motion-SPADE model. them in terms of identity and expression preservation, on a scale from 1 to 5. In a first round of evaluations, we display side by side the four main dense models results, and in the second round, Motion Net and Motion-SPADE results using six different mobile architectures. We collect in each case 500 pairwise evaluations, each from 5 different participants. Dense approaches results present a large variability and human scores seem to agree with metrics. Mobile models results differences are more subtle, but the Hybrid Motion-SPADE using 10 landmarks model seems preferred.

Figure D-1 illustrates the quality performance reached on mobile. The quality of results degrades in presence of large head rotations. Still, the Motion-SPADE results are visually close to the targets, particularly it renders lips and teeth better. The H264 compression results are displayed given a bandwidth of 9 kbit/s, to be compared to the ones of the Motion-SPADE 20 model that runs the fastest on mobile. At this bandwidth, video transmission is hardly possible using standard codecs.

On-device Real-time Inference

We develop a standalone app that is able to perform a real-time call between two participants. In order to do so, we converted the models to int8, then to TorchScript and run them on the phone's CPU. To compress the Motion based models, we only rely on int8 since the non-compressed models are already small. We use WebRTC to also feature the sound during the call to make the experience more immersive. Screenshots of the app are displayed in Figure 7-4 Average scores (Higher is better) with confidence intervals.

Conclusion

Our exploration of state-of-the art animation models led us to the following observations: The Neural Talking head results are qualitatively satisfactory, but the fine-tuning step requirement makes the approach complex to implement in practice. Using a full face segmentation approach seems unfit to a low bandwidth application.

The Bilayer approach and the FOM methods perform best towards reaching a correct low-bandwidth/quality trade-off on mobile. Our human study shows that FOM results are preferred in terms of identity and expression preservation. Focusing on this best candidate approach, we design a novel hybrid architecture taking advantage of the high fidelity to the target thanks to the warping principle, and enhancing the quality of important attributes with SPADE blocks. Only exploiting polygons induced segments with this approach improves quality without high transmission cost.

The obtained image quality is close to the one reached by dense models while running in real-time on Mobile CPU. The bandwidth required to send a video is lower than the one required for sending audio. There are a number of interesting challenges to tackle next to improve quality of the generations, e.g. generating large head rotation movements, hands, or using pupils tracking.

Chapter 8

Discussion

In this Chapter, we briefly summarize our contributions and sketch some interesting future research directions.

Summary of Contributions

In this manuscript, we presented our contributions on the redundancy and efficiency of Neural Networks in ascending order of applicability.

Equivalence Classes

In Chapter 3, we studied the functional equivalence classes of Neural Networks from a local perspective in the space of parameters for networks with arbitrary depth, relying on our standalone characterization for one-hidden-layer networks along with algebraic and geometrical tools that we designed. This allows us to group networks that behave similarly by orbits under the action of the rescaling the permutation operations. Using these theoretical considerations, we developed an alternative to SGD that operates in the quotient space in Chapter 4, called ENorm. The method selects the representant of the current orbit that minimizes a certain energy after each SGD step. This work aiming at properly balancing the energy of the network was later leveraged by [START_REF] Nagel | Data-free quantization through weight equalization and bias correction[END_REF] to design a robust int8 quantization scheme that equalizes the weights before compression.

Neural Network Compression

In Chapter 5, we designed a product quantization method, called iPQ, that allows us to drastically reduce the size of a network while almost preserving its accuracy. This method was improved by [START_REF] Martinez | Permute, quantize, and fine-tune: Efficient compression of neural networks[END_REF], who also learn an adequate permutation of the weights to enforce better redundancy when considering large quantization blocks. Next, aiming at applying iPQ to any architecture, we developed a preconditioning method called Quant-Noise that injects carefully selected quantization noise when training the network before its compression in Chapter 6. Quant-Noise yields competitive results for scalar quantization techniques such as int8.

Low-Bandwidth Video Chat

The compressed size of the network is a significant indicator of the quality of the quantization. However, other metrics such as inference time and battery usage are also relevant, especially for on-device, real time applications. In order to confront our findings to real-world constraints, we developed a low-bandwidth generative video chat called FaceGen in Chapter 7. FaceGen generates the face on the receiver side using a GAN conditioned on a stream of landmarks plus an identity embedding sent at the beginning of the call. FaceGen models weight a total of less than 2 MB, run in real-time on an iPhone 8 and compress the video bandwidth to less than 10 kbits/s.

Future Directions

We list below some directions for further research on the short and longer run.

Equivalence Classes

Equivalence classes allow us to conveniently operate in the quotient space of neural networks, provided that we know the right representant to manipulate.

Canonical Representant. To properly benefit from the theoretical advances regarding the functional equivalence classes, the question of the choice of the adequate representant is key. For instance, with ENorm in Chapter 4, we interleave SGD updates with a step that selects the representant of the current orbit that minimizes the 𝐿 2 global norm of its weights. While this particular choice leads to a fast iterative algorithm and presents empirical evidence of its pertinence, this criterion is not theoretically grounded. One possibility would be to select the adequate representant such that the next SGD step is equivalent to a second order natural gradient update.

Non-local Characterization. On the more theoretical side on the characterization of the functional equivalence classes, we wish to reconcile the sufficient condition for restricted local identifiability and the necessary condition for local identifiability in Chapter 3 to properly characterize locally identifiable parameterizations. Next, we argue that having a global characterization would necessitate to take the permutations into account and to design a more general way to modify 𝜃 as follows: keep the multiplicative updates of the form 𝑤𝑒 𝛾 for non-zero weights 𝑤 or biases 𝑏 and default back to additive updates of the form 𝑤 + 𝛿 for null parameters.

Neural Network Compression

Compression and quantization is a major challenge for the research industry in order to deploy deep learning models on-device and for more privacy.

Deployment of iPQ. First, we hope that our iPQ + int8 implementation and open-sourced code (see Chapters 5 and 6) will unlock on-device model deployment for models with 10 to 100 million parameters. Such models would enable model deployment off-the-shelf as follows. First, compress then model with iPQ + int8 and send it on-device. Then, decompress the iPQ part on-device to default back to a plain int8 model to benefit from the speedup of int8. Further research directions include designing our own kernels to perform the forward pass in the iPQ + int8 domain. This would imply to pre-compute the dot products between activations and the codebook to build and store Look-Up Tables (LUTs) in low-level cache memory and then lookup and sum the results. Larger block sizes, fewer codewords and larger underlying matrices would help reduce the latency.

Compression as a Service. One of the current limitations of iPQ is that the number of codewords and the block size has to be manually tuned using in-domain knowledge. For instance, we observed that the classifiers in convolutional networks are harder to compress and therefore use a block size of 4 instead of 8 and/or more codewords for this particular layer. Conversely, some methods (see Subsection 2.3.5) for mixed scalar quantization propose to learn the precision -the number of bitsper layer in a fully differentiable way. One interesting direction would be to apply such a paradigm to iPQ to foster its adoption. One orthogonal direction would be to derive heuristics or methods to select the right compression method for a specific task, target metrics and hardware. Currently, as explained in Section 2.3, the current process is to tailor one of the many available methods listed in Section 2.3 -or a combination thereof -to a specific task by trials-and-errors. For instance, distillation and then iPQ proved to work well in that order. Adaptability would be a determinant factor for the broad adoption of compression methods.

Training Bigger Models. The next generation of Nvidia's GPUs, the A100, now natively supports int8 and int4 additionally to half-precision training in fp16. To benefit from this theoretical speedup, we could train models with more parameters than currently done, provided that we are able to design a training recipe to preserve the accuracy of the network. Hence, advances in both the hardware and the training techniques could help taking neural networks to the next level.

Low-Bandwidth Video Chat

While incremental changes in the current method would make face generation more convincing, such methods also pave the way for 3D face generation.

Incremental changes. There are many possible incremental ameliorations to the current setup of Chapter 7. For instance, we could improve the data loaders to sample more large head rotations since the model performs currently poorly on these. We could also be more strict when sampling the reference frame and have a neutral expression with a person facing the camera. In the search for more efficiency, we could also incorporate new blocks such as FBNetV3 blocks [START_REF] Dai | Fbnetv3: Joint architecture-recipe search using neural acquisition function[END_REF] and optimize some operations performed in the Dense Motion network such as the grid sample.

3D Face Reconstruction. On the longer term, similar insights could be applied to 3D face reconstruction, with or without the low bandwidth constraint. The idea is to leverage (1) the growing availability of augmented and virtual reality devices and (2) the desire of people to stay close to their relative ones that live remotely. On traditional devices, the 3D face could be projected back to 2D images, while on more advanced VR/AR headsets such as the Oculus Quest 2, this could provide an immersive and realistic experience of remote presence.

𝑊 𝐿 = ( 𝐷 (𝐿-1)
) -1 𝑊 (𝐿) , we have

𝑤 ′ 𝜇→𝜈 = 𝑤 𝜇→𝜈 /𝛾 𝜇 . (A.4)
Finally, regarding the biases, 𝑏 ′ (ℓ) = 𝑏 (ℓ) 𝐷 (ℓ) for every layer 1 ≤ ℓ ≤ 𝐿 gives, for every hidden neuron 𝜈 ∈ 𝐻, 𝑏 ′ 𝜈 = 𝛾 𝜈 𝑏 𝜈 . Thus, defining 𝑠 as in representation (2.8), we obtain 𝑠(𝜃) = 𝜃 ′ . Hence, 𝜃 ∼ Neuron 𝜃 ′ .

Neuron =⇒ Path. If 𝜃 ∼ 2 𝜃 ′ , there exists 𝑠 as in representation (2.8) such that 𝜃 ′ = 𝑠(𝜃). First, we deduce that sign(𝜃) = sign(𝜃 ′ ). Next, let 𝑝 = (𝜈 0 , . . . , 𝜈 𝐿 ) ∈ 𝒫 be a full path. Then,

𝑣 𝑝 (𝜃 ′ ) = 𝑤 ′ 𝜈 0 →𝜈 1 𝑤 ′ 𝜈 1 →𝜈 2 . . . 𝑤 ′ 𝜈 𝐿-1 →𝜈 𝐿 = ( 𝛾 𝜈 1 𝑤 𝜈 0 →𝜈 1 )( 𝛾 𝜈 2 𝛾 𝜈 1 𝑤 𝜈 1 →𝜈 2 ) . . . ( 1 𝛾 𝜈 𝐿 𝑤 ′ 𝜈 𝐿-1 →𝜈 𝐿 ) = 𝑤 𝜈 0 →𝜈 1 𝑤 𝜈 1 →𝜈 2 . . . 𝑤 𝜈 𝐿-1 →𝜈 𝐿 = 𝑣 𝑝 (𝜃). (A.5) (A.6) (A.7) (A.8)
Similarly, we show that for every partial path 𝑞 ∈ 𝒬, 𝑏 𝑞 0 𝑣 𝑞 (𝜃) = 𝑏 ′ 𝑞 0 𝑣 𝑞 (𝜃 ′ ).

Path =⇒ Trajectory. If 𝜃 ∼ Path 𝜃 ′ , since sign(𝜃 ′ ) = sign(𝜃), using Proposition 3.3.5, there exists a unique 𝛾 = (𝛼, S𝛽) ∈ Supp 𝜃 such that 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 . Let us show that 𝛾 is 𝜃-rescaling-compatible. Let us show first that 𝛼 ∈ ker (P 𝜃 ). Since (P 𝜃 𝛼) 𝑝 = 1(𝑣 𝑝 (𝜃) ̸ = 0)P𝛼, it suffices to show that, for every full path 𝑝 ∈ 𝒫 such that 𝑣 𝑝 (𝜃) ̸ = 0, we have (P𝛼) 𝑝 = 0. We compute

𝑣 𝑝 (𝜃 ′ ) = 𝑣 𝑝 (𝜃) ∏ 𝑒∈𝑝 exp(𝛼 𝑒 ) = 𝑣 𝑝 (𝜃) exp ⎛ ⎝ ∑ 𝑒∈𝑝 𝛼 𝑒 ⎞ ⎠ = 𝑣 𝑝 (𝜃) exp((P𝛼) 𝑝 ). (A.9)
Using the fact that 𝑣 𝑝 (𝜃 ′ ) = 𝑣 𝑝 (𝜃), we deduce that (P𝛼) = 0. Thus, 𝛼 ∈ ker (P 𝜃 ) 𝑝 .

Next, let us similarly show that 𝛼 -𝛽 ∈ ker (Q 𝜃 ). Since (Q 𝜃 𝛼) 𝑞 = 1(𝑏 𝑞 0 𝑣 𝑞 (𝜃) ̸ = 0)P𝛼, it suffices to show that, for every partial path 𝑞 ∈ 𝒬 such that 𝑏 𝑞 0 𝑣 𝑞 (𝜃), we have (Q𝛼) 𝑞 = 0. We compute

𝑏 ′ 𝑞 0 𝑣 𝑞 (𝜃 ′ ) = 𝑏 𝑞 0 𝑣 𝑞 (𝜃) exp (S𝛽 𝑞 0 ) exp ⎛ ⎝ ∑ 𝑒∈𝑞 𝛼 𝑒 ⎞ ⎠ .
Recalling that the definition of S𝛽 𝑞 0 is independent of the choice of the path going from 𝑞 0 to any output neuron (see Proposition 3.3.6 since 𝛽 ∈ ker P),

𝑏 ′ 𝑞 0 𝑣 𝑞 (𝜃 ′ ) = 𝑏 𝑞 0 𝑣 𝑞 (𝜃) exp ⎛ ⎝ - ∑ 𝑒∈𝑞 𝛽 𝑒 ⎞ ⎠ ⎛ ⎝ ∑ 𝑒∈𝑞 𝛼 𝑒 ⎞ ⎠ = 𝑏 𝑞 0 𝑣 𝑞 (𝜃) exp ⎛ ⎝ ∑ 𝑒∈𝑞 (𝛼 𝑒 -𝛽 𝑒 ) ⎞ ⎠ = 𝑏 𝑞 0 𝑣 𝑞 (𝜃) exp ( (Q(𝛼 -𝛽)) 𝑞 )
.

Using the fact that 𝑏 ′ 𝑞 0 𝑣 𝑝 (𝜃 ′ ) = 𝑏 𝑞 0 𝑣 𝑞 (𝜃), we deduce that (Q(𝛼 -𝛽)) 𝑞 = 0. Thus, 𝛼𝛽 ∈ ker (Q 𝜃 ). Thus, we have proven that 𝜃 ∼ Trajectory 𝜃 ′ . • If 𝜈 ∈ 𝑁 𝐿 , let us show that 𝑤 ′ 𝑒 = 𝑤 𝑒 /𝛿 𝜇 𝑒 ∈ 𝐸 such that 𝑤 𝑒 ̸ = 0. Since 𝑞 = (𝜇, 𝜈) is a path going from 𝜇 to the output neuron such that 𝑣 𝑞 (𝜃) ̸ = 0, we have (S𝛼) 𝜇 = -𝛼 𝑒 . Thus, 𝑤 ′ 𝑒 = 𝑤 𝑒 exp(𝛼 𝑒 ) = 𝑤 𝑒 exp((S𝛼) 𝜈 ) = 𝑤 𝑒 /𝛿 𝜈 .

Trajectory

• Otherwise, let us show that 𝑤 ′ 𝑒 = 𝑤 𝑒 𝛿 𝜈 /𝛿 𝜇 for every 𝑒 ∈ 𝐸 such that 𝑤 𝑒 ̸ = 0. Let 𝑞 be a partial path going from neuron 𝜈 to the output neuron such that 𝑣 𝑞 (𝜃) ̸ = 0 and let 𝑞 = (𝜇) + 𝑞. We observe that 𝑣 𝑞 (𝜃) ̸ = 0 and that

𝛼 𝑒 = ∑ 𝑒 ′ ∈𝑞 𝛼 𝑒 ′ - ∑ 𝑒 ′ ∈𝑞 𝛼 𝑒 ′ = (S𝛼) 𝜈 -(S𝛼) 𝜇 (A.15) Thus, 𝑤 ′ 𝑒 = 𝑤 𝑒 exp(𝛼 𝑒 ) = 𝑤 𝑒 exp((S𝛼) 𝜈 -(S𝛼) 𝜇 ) = 𝑤 𝑒 𝛿 𝜈 /𝛿 𝜇 .
Thus, for all 1 ≤ ℓ ≤ 𝐿, 𝑊

′ (ℓ) = ( 𝐷 (ℓ-1)
) -1 𝑊 (ℓ) 𝐷 (ℓ) and 𝑏 ′ (ℓ) = 𝑏 (ℓ) 𝐷 (ℓ) . Therefore, 𝜃 ′ ∼ Layer 𝜃, which concludes the proof.

A.1.2 Link with Functional Equivalence

We formally prove separately that the permutation and rescaling operations preserve the function implemented by the network. Next, we show that permutation and rescaling operations commute, which allows us to define permutation-rescaling equivalent parameters. Finally, we show that permutation-rescaling equivalence implies functional equivalence. For the last layer, as 𝜋 (𝐿) = 𝐼 𝑁 𝐿 , 𝑦 (𝐿) (𝜃 ′ , 𝑥) = 𝑦 (𝐿-1) (𝜃 ′ , 𝑥)𝑊 ′(𝐿) + 𝑏 ′(𝐿) similarly yields

𝑦 (𝐿) (𝜃 ′ , 𝑥) = 𝑦 (𝐿) (𝜃, 𝑥) thus 𝑅 𝐺 (𝜃) = 𝑅 𝐺 (𝜃 ′ ).
Proof of Proposition A.1.3

Proposition. Two parameterizations 𝜃 and 𝜃 ′ are permutation-rescaling equivalent iff they are rescaling-permutation equivalent.

Proof. Let 𝜃 and 𝜃 ′ be permutation-rescaling equivalent parameterizations, with no biases. There exists a parameterization 𝜃 ′′ such that 𝜃 ∼ 𝑃 𝜃 ′′ and 𝜃 ′′ ∼ 𝑆 𝜃 ′ . Thus, for every ℓ ∈ 1, 𝐿 ,

𝑊 ′′ (ℓ) = ( 𝜋 (ℓ-1) ) -1 𝑊 (ℓ) 𝜋 (ℓ) 𝑊 ′ (ℓ) = ( 𝐷 (ℓ-1) ) -1 𝑊 ′′(ℓ) 𝐷 (ℓ) = ( 𝐷 (ℓ-1) ) -1 ( 𝜋 (ℓ-1) ) -1 𝑊 (ℓ) 𝜋 (ℓ) 𝐷 (ℓ) = ( 𝜋 (ℓ-1) ) -1 ( ̃ 𝐷 (ℓ-1) ) -1 𝑊 (ℓ) ̃ 𝐷 (ℓ) 𝜋 (ℓ) (A.26) (A.27) (A.28) (A.29)
where, for every ℓ ∈ 0, 𝐿 ,

̃ 𝐷 (ℓ) = 𝜋 (ℓ) 𝐷 (ℓ) ( 𝜋 (ℓ) ) -1 (A.30)
We verify that ̃ 𝐷 (ℓ) is a diagonal matrix with strictly positive entries, 𝐷 (ℓ) ∈ 𝒟(|𝑁 ℓ |). We define

𝑊 ′′′ (ℓ) = ( ̃ 𝐷 (ℓ-1) ) -1 𝑊 (ℓ) ̃ 𝐷 (ℓ) (A.31)
Using Equation (A.29), we have

𝑊 ′(ℓ) = ( 𝜋 (ℓ-1) ) -1 𝑊 ′′′(ℓ) 𝜋 (ℓ) (A.32)
𝑤 𝜈→𝜂 𝜎(⟨𝑤 •→𝜈 , 𝑥⟩ + 𝑏 𝜈 ) + 𝑏 𝜂 ⏟ ⏞ 𝜙𝜂(𝑥) = ∑ 𝜈∈𝑁 1 𝑤 ′ 𝜈→𝜂 𝜎(⟨𝑤 ′ •→𝜈 , 𝑥⟩ + 𝑏 ′ 𝜈 ) + 𝑏 ′ 𝜂 ⏟ ⏞ 𝜓𝜂(𝑥) . (A.33)
The proof follows four steps:

1. We reduce to the scalar case by taking 𝑥 = 𝑢𝑡 + 𝑟, 𝑡 ∈ R in Equation (A.33) for carefully chosen sets of directions 𝒰 ⊂ R 𝑁 0 and of offsets ℛ 𝑢 ⊂ R 𝑁 0 (note that ℛ 𝑢 depends on 𝑢 ∈ 𝒰). We choose these sets such that, for all 𝑢 ∈ 𝒰 and 𝑟 ∈ ℛ 𝑢 , and for any output neuron 𝜂 ∈ 𝑁 2 , the breakpoints of the continuous piecewise affine scalar function 𝑡 ↦ → 𝜙 𝜂 (𝑢𝑡 + 𝑟) are distinct, and similarly for 𝜓 𝜂 .

2. We fix one output neuron 𝜂 and we fix 𝑢 ∈ 𝒰, 𝑟 ∈ ℛ 𝑢 . We use the identifiability lemma A.1.1 on the functions 𝑡 ↦ → 𝜙 𝜂 (𝑢𝑡 + 𝑟) and 𝑡 ↦ → 𝜓 𝜂 (𝑢𝑡 + 𝑟) (that depend on both 𝜂 and 𝑢) to (partially) identify the weights and biases of 𝜃 and 𝜃 ′ .

3. We prove that the (partial) identification of the weights and biases of 𝜃 and 𝜃 ′ obtained in Step 2 is independent from 𝑢 and 𝑟. It allows to define the rescalings and the permutation of the hidden neurons with respect to the output neuron 𝜂. Note that such rescalings and permutation still depend a priori on 𝜂.

Step 1. We first construct 𝒰 and then, for every 𝑢 ∈ 𝒰, we construct ℛ 𝑢 .

• Recall that none of the vectors 𝑤 •→𝜈 and 𝑤 ′ •→𝜈 equals zero for 𝜈 ∈ 𝑁 1 . Then, we define 𝒰 0 as R 𝑁 0 minus a finite union of hyperplanes as follows:

𝒰 0 = R 𝑁 0 ∖ ⎛ ⎝ ⋃ 𝜈∈𝑁 1 𝑤 ⊥ •→𝜈 ∪ 𝑤 ′ ⊥ •→𝜈 ⎞ ⎠ (A.34)
where

𝑤 ⊥ •→𝜈 = {𝑢 ∈ R 𝑁 0 | ⟨𝑢, 𝑤 •→𝜈 ⟩ = 0}.
In particular, for all 𝑢 ∈ 𝒰 0 , we have

∀𝜈 ∈ 𝑁 1 , ⟨𝑤 •→𝜈 , 𝑢⟩ ̸ = 0, ∀𝜈 ∈ 𝑁 1 , ⟨𝑤 ′ •→𝜈 , 𝑢⟩ ̸ = 0. (A.35) (A.36) Next, let 𝜂 ∈ 𝑁 2 and define ℐ 𝜂 = {𝜈 ∈ 𝑁 1 | 𝑤 𝜈→𝜂 ̸ = 0}. Let 𝒥 ⊂ ℐ 𝜂 and define 𝐻 𝒥 ,𝜂 = { 𝑢 ∈ R 𝑁 0 ⃒ ⃒ ⃒ ⟨ 𝑢, ∑ 𝜈∈𝒥 𝑤 𝜈→𝜂 𝑤 •→𝜈 ⟩ = 0 } . (A.37)
Since 𝜃 is irreducible, there are no co-dependant hidden neurons (Definition 3.2.3), hence ∑ 𝜈∈𝒥 𝑤 𝜈→𝜂 𝑤 •→𝜈 ̸ = 0, hence 𝐻 𝒥 ,𝜂 is a hyperplane of dimension 𝑁 0 -1. Since there are finitely many sets 𝒥 , we define

𝒰 1 = R 𝑁 0 ∖ ( ⋃ 𝜂∈𝑁 2 𝒥 ⊂ℐ𝜂 𝐻 𝒥 ,𝜂 ) . (A.38)
Hence, for all 𝑢 ∈ 𝒰 1 , we have

∀𝜂 ∈ 𝑁 2 , ∀𝒥 ⊂ ℐ 𝜂 , ⟨ 𝑢, ∑ 𝜈∈𝒥 𝑤 𝜈→𝜂 𝑤 •→𝜈 ⟩ ̸ = 0. (A.39)
We finally define 𝒰 = 𝒰 0 ∪ 𝒰 1 .

• We fix 𝑢 ∈ 𝒰 and construct ℛ 𝑢 such that, for all 𝜈 1 ̸ = 𝜈 2 ∈ 𝑁 1 and 𝑟 ∈ ℛ 𝑢 ,

𝑏 𝜈 1 + ⟨𝑤 •→𝜈 1 , 𝑟⟩ ⟨𝑤 •→𝜈 1 , 𝑢⟩ ̸ = 𝑏 𝜈 2 + ⟨𝑤 •→𝜈 2 , 𝑟⟩ ⟨𝑤 •→𝜈 2 , 𝑢⟩ , 𝑏 ′ 𝜈 1 + ⟨𝑤 ′ •→𝜈 1 , 𝑟⟩ ⟨𝑤 ′ •→𝜈 1 , 𝑢⟩ ̸ = 𝑏 ′ 𝜈 2 + ⟨𝑤 ′ •→𝜈 2 , 𝑟⟩ ⟨𝑤 ′ •→𝜈 2 , 𝑢⟩ . (A.40) (A.41) Equation (A.40) writes ⟨𝑟, 𝛼 𝜈 1 ,𝜈 2 ⟩ + 𝛽 𝜈 1 ,𝜈 2 ̸ = 0, where 𝛼 𝜈 1 ,𝜈 2 = 𝑤 •→𝜈 1 ⟨𝑤 •→𝜈 2 , 𝑢⟩ -𝑤 •→𝜈 2 ⟨𝑤 •→𝜈 1 , 𝑢⟩ 𝛽 𝜈 1 ,𝜈 2 = 𝑏 𝜈 1 ⟨𝑤 •→𝜈 2 , 𝑢⟩ -𝑏 𝜈 2 ⟨𝑤 •→𝜈 1 , 𝑢⟩. (A.42) (A.43)
Since 𝜃 is irreducible, there are no twin neurons (Definition 3.2.2). Define, for any

𝜈 1 ̸ = 𝜈 2 ∈ 𝑁 1 , the set 𝐻 𝜈 1 ,𝜈 2 = { 𝑟 ∈ R 𝑁 0 | ⟨𝑟, 𝛼 𝜈 1 ,𝜈 2 ⟩ + 𝛽 𝜈 1 ,𝜈 2 = 0
} which is either the empty set, a hyperplane, or the whole space R 𝑁 0 . Let us show that 𝐻 𝜈 1 ,𝜈 2 is not equal to R 𝑁 0 . For the sake of contradiction, let us assume that 𝐻 𝜈 1 ,𝜈 2 = R 𝑁 0 . Then, both 𝛼 𝜈 1 ,𝜈 2 = 0 and 𝛽 𝜈 1 ,𝜈 2 = 0. Using Equations (A.42) and (A.43), we deduce that1 𝑤

•→𝜈 1 = 𝑑𝑤 •→𝜈 2 and 𝑏 𝜈 1 = 𝑑𝑏 𝜈 2 with 𝑑 = ⟨𝑤 •→𝜈 1 , 𝑢⟩/⟨𝑤 •→𝜈 2 , 𝑢⟩ ̸ = 0, which is equivalent to Γ 𝜈 1 = Γ 𝜈 2 in Defini- tion 3.2.2,
which is a contradiction since there are no hidden twin neurons in 𝜃. Thus, we have shown that 𝐻 𝜈 1 ,𝜈 2 is either an affine hyperplane, or the empty set. We similarly define 𝐻 ′ 𝜈 1 ,𝜈 2 and proceed to the same reasoning with 𝜃 ′ .

Finally, we define ℛ 𝑢 as a R 𝑁 0 minus a finite union of hyperplanes as follows.

ℛ 𝑢 = R 𝑁 0 ∖ ⎛ ⎝ ⋃ 𝜈 1 ̸ =𝜈 2 ∈𝑁 1 𝐻 𝜈 1 ,𝜈 2 ∪ 𝐻 ′ 𝜈 1 ,𝜈 2 ⎞ ⎠ . (A.44)
Note that by construction, for all 𝜈 1 ̸ = 𝜈 2 ∈ 𝑁 1 and 𝑟 ∈ ℛ 𝑢 , Equations (A.40) and (A.41) are satisfied.

Step 2. We fix an output neuron 𝜂 ∈ 𝑁 2 and define ℐ

= {𝜈 ∈ 𝑁 1 | 𝑤 𝜈→𝜂 ̸ = 0} and ℐ ′ = {𝜈 ∈ 𝑁 1 | 𝑤 ′ 𝜈→𝜂 ̸ = 0}
. Next, we fix 𝑢 ∈ 𝒰 and 𝑟 ∈ ℛ 𝑢 . Using Equation (A.33), we have that for all 𝑡 ∈ R, 𝜙 𝜂 (𝑢𝑡 + 𝑟) = 𝜓 𝜂 (𝑢𝑡 + 𝑟). Then, for all 𝜈 ∈ 𝑁 1 , we define

𝑠 𝜈 = sign(⟨𝑤 •→𝜈 , 𝑢⟩) 𝑎 𝜈 = 𝑤 𝜈→𝜂 ⟨𝑤 •→𝜈 , 𝑢⟩𝑠 𝜈 𝑡 𝜈 = -(𝑏 𝜈 + ⟨𝑤 •→𝜈 , 𝑟⟩)/⟨𝑤 •→𝜈 , 𝑢⟩ 𝑐 = 𝑏 𝜂 (A.45) (A.46) (A.47) (A.48)
and we similarly define 𝑎 ′ 𝜈 , 𝑠 ′ 𝜈 , 𝑡 ′ 𝜈 and 𝑐 ′ 𝜈 . Note that these coefficients depend on 𝑢 and 𝑟, although we omit to state the dependency explicitly for the sake of clarity. We verify that the following three conditions are met.

• For all 𝜈 ∈ ℐ, 𝑎 𝜈 ̸ = 0. Indeed, by definition of ℐ, 𝑤 𝜈→𝜂 ̸ = 0 and by Equation (A.35), we have that ⟨𝑤 •→𝜈 , 𝑢⟩ ̸ = 0. Similarly, for all 𝜈 ∈ ℐ ′ , we have

𝑎 ′ 𝜈 ̸ = 0.
• For all 𝜈 1 ̸ = 𝜈 2 ∈ ℐ, 𝑡 𝜈 1 ̸ = 𝑡 𝜈 2 . This comes from Equations (A.40) and (A.41).

Similarly, for all

𝜈 1 ̸ = 𝜈 2 ∈ ℐ ′ , 𝑡 ′ 𝜈 1 ̸ = 𝑡 ′ 𝜈 2 .
• For all 𝒥 ⊂ ℐ, We get 𝑏 𝜂 = 𝑏 ′ 𝜂 and ℐ = ℐ ′ up to a re-numbering. We distinguish between two cases. If ℐ = ∅ and ℐ ′ = ∅, we directly proceed to Step 3. Otherwise, ℐ ̸ = ∅ and ℐ ′ ̸ = ∅. We explicitly denote 𝜋 : ℐ ↦ → ℐ ′ the re-numbering of ℐ ′ . Then, for all 𝜈 ∈ ℐ,

𝑠 ′ 𝜋(𝜈) = sign(⟨𝑤 ′ •→𝜋(𝜈) , 𝑢⟩) = sign(⟨𝑤 •→𝜈 , 𝑢⟩) = 𝑠 𝜈 𝑤 ′ 𝜋(𝜈)→𝜂 ⟨𝑤 ′ •→𝜋(𝜈) , 𝑢⟩𝑠 ′ 𝜋(𝜈) = 𝑤 𝜈→𝜂 ⟨𝑤 •→𝜈 , 𝑢⟩𝑠 𝜈 -(𝑏 ′ 𝜋(𝜈) + ⟨𝑤 ′ •→𝜋(𝜈) , 𝑟⟩)/⟨𝑤 ′ •→𝜋(𝜈) , 𝑢⟩ = -(𝑏 𝜈 + ⟨𝑤 •→𝜈 , 𝑟⟩)/⟨𝑤
′ (𝑡 + 𝑖 ) -𝜙 ′ (𝑡 - 𝑖 ) = 𝜓 ′ (𝑡 + 𝑖 ) -𝜓 ′ (𝑡 - 𝑖 ) to obtain 𝑎 𝑖 𝑠 𝑖 (1(𝑠 𝑖 ) -1(-𝑠 𝑖 )) + 𝑓 (𝑡 + 𝑖 ) -𝑓 (𝑡 - 𝑖 ) = 𝑎 ′ 𝑖 𝑠 ′ 𝑖 (1(𝑠 ′ 𝑖 ) -1(-𝑠 ′ 𝑖 )) + 𝑔(𝑡 + 𝑖 ) -𝑔(𝑡 - 𝑖 )
Using the facts that 𝑓 (𝑡 + 𝑖 ) = 𝑓 (𝑡 - 𝑖 ) and 𝑔(𝑡 + 𝑖 ) = 𝑔(𝑡 - 𝑖 ), this simplifies to

𝑎 𝑖 𝑠 𝑖 (1(𝑠 𝑖 ) -1(-𝑠 𝑖 )) = 𝑎 ′ 𝑖 𝑠 ′ 𝑖 (1(𝑠 ′ 𝑖 ) -1(-𝑠 ′ 𝑖 )), (A.71) which finally yields 𝑎 𝑖 𝑠 2 𝑖 = 𝑎 ′ 𝑖 𝑠 ′ 2 𝑖 .
Since 𝑠 𝑖 ∈ {-1, +1}, we deduce that 𝑠 2 𝑖 = 1, and similarly 𝑠 ′ 2 𝑖 = 1. Hence, 𝑎 𝑖 = 𝑎 ′ 𝑖 and there exists 𝜀 𝑖 ∈ {-1, +1} such that 𝑠 ′ 𝑖 = 𝜀 𝑖 𝑠 𝑖 . We then re-write the equality 𝜙(𝑡) = 𝜓(𝑡) as follows:

∑ 𝑖∈ℐ 𝑎 𝑖 [ 𝜎(𝑠 𝑖 (𝑡 -𝑡 𝑖 )) -𝜎(𝜀 𝑖 𝑠 𝑖 (𝑡 -𝑡 𝑖 )) ] + 𝑐 -𝑐 ′ = 0. (A.72)
Next, we observe that

𝜎(𝑠 𝑖 (𝑡 -𝑡 𝑖 )) -𝜎(𝜀 𝑖 𝑠 𝑖 (𝑡 -𝑡 𝑖 )) = ⎧ ⎪ ⎨ ⎪ ⎩ 0 if 𝜀 𝑖 = 1 𝑠 𝑖 (𝑡 -𝑡 𝑖 ) if 𝜀 𝑖 = -1 (
𝑅 𝜃 (𝑥) 𝜈 = ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) + 𝐿 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ 𝑞 ℓ =𝜈 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥).
(A.75)

Proof. We prove the result by induction on the number of layers 𝐿.

• If 𝐿 = 0, the input and output layers are the same and 𝑅 𝜃 (𝑥) = 𝑥. On the other hand, 𝒫 = {(𝜇)} 𝜇∈𝑁 0 . Since there are no non-linearities, 𝑎 𝑝 (𝜃, 𝑥) = 1 for all 𝑝, 𝑥. We also have 𝑣 𝑝 (𝜃) = 1 for all 𝑝. It follows that

∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) = ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 = 𝑥 𝜈 (A.76)
• If 𝐿 = 1, there exists 𝑊 and 𝑏 such that 𝑅 𝜃 (𝑥) = 𝑥𝑊 + 𝑏. On the other hand, 𝒫 = {(𝜇, 𝜈)} 𝜇∈𝑁 0 ,𝜈∈𝑁 1 and 𝒬 1 = {(𝜈)} 𝜈∈𝑁 1 . Since there are no non-linearities, 𝑎 𝑞 (𝜃, 𝑥) = 1 for all 𝑞 ∈ 𝒫 ∪ 𝒬 1→1 . We also have 𝑣 𝑝 (𝜃) = 𝑤 𝜇→𝜈 for all 𝑝 = (𝜇, 𝜈). It follows that

∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) = ∑ 𝜇∈𝑁 0 𝑥 𝑝 0 𝑤 𝜇→𝜈 = (𝑥𝑊 ) 𝜈 (A.77) and 𝐿 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ 𝑞 ℓ =𝜈 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥) = ∑ 𝑞∈𝒬 1 𝑞 0 =𝜈 𝑏 𝑞 0 = 𝑏 𝜈 (A.78) • If 𝐿 ≥ 2, for any 𝑥 ∈ R 𝑁 0 and any 𝜈 ∈ 𝑁 𝐿 , 𝑅 𝜃 (𝑥) 𝜈 = ∑ 𝜇∈𝑁 𝐿-1 𝑤 𝜇→𝜈 𝑦 𝜇 (𝜃, 𝑥) + 𝑏 𝜈 = ∑ 𝜇∈𝑁 𝐿-1 𝑤 𝜇→𝜈 𝜎 ( ̃ 𝑅 𝜃 (𝑥) 𝜇 ) + 𝑏 𝜈 (A.79) (A.80)
where ̃ 𝑅 𝜃 (𝑥) denotes the output of layer 𝐿 -1 of the network 𝐺 valued with 𝜃, evaluated before the non-linearity. Using the induction hypothesis, we get, for

𝜇 ∈ 𝑁 𝐿-1 ,  𝑅 𝜃 (𝑥) 𝜇 = ∑ p∈ ̃ 𝒫 p𝐿-1 =𝜇 𝑥 p0 𝑣 p(𝜃)𝑎 p(𝜃, 𝑥) + 𝐿-1 ∑ ℓ=1 ∑ q∈ ̃ 𝒬 ℓ q𝐿-1 =𝜇 𝑏 q0 𝑣 q(𝜃)𝑎 q(𝜃, 𝑥) (A.
𝑅 𝜃 (𝑥) 𝜈 = ∑ 𝜇∈𝑁 𝐿-1 𝑤 𝜇→𝜈 1(𝑦 𝜇 (𝜃, 𝑥) > 0) ̃ 𝑅 𝜃 (𝑥) 𝜇 + 𝑏 𝜈 = ∑ 𝜇∈𝑁 𝐿-1 ∑ p∈ ̃ 𝒫 p𝐿-1 =𝜇 𝑥 p0 𝑤 𝜇→𝜈 𝑣 p(𝜃)1(𝑦 𝜇 (𝜃, 𝑥) > 0)𝑎 p(𝜃, 𝑥) + ∑ 𝜇∈𝑁 𝐿-1 𝐿-1 ∑ ℓ=1 ∑ q∈ ̃ 𝒬 ℓ q𝐿-1 =𝜇 𝑏 q0 𝑤 𝜇→𝜈 𝑣 q(𝜃)1(𝑦 𝜇 (𝜃, 𝑥) > 0)𝑎 q(𝜃, 𝑥) + 𝑏 𝜈 (A.83) (A.84)
We next simplify Equation (A.84) using the following remarks. Recall that we denote path concatenation with +. Let 𝑞 = q + 𝜇 and 𝑝 = p + 𝜇. First, 𝑤 𝜇→𝜈 𝑣 p(𝜃) = 𝑣 𝑝 (𝜃) by definition of 𝑣 𝑝 (𝜃) and similarly for 𝑞. Second, 1(𝑦 𝜇 (𝜃, 𝑥) > 0)𝑎 p(𝜃, 𝑥) = 𝑎 𝑝 (𝜃, 𝑥) by definition of 𝑎 𝑝 (𝜃, 𝑥) and similarly for 𝑞. Finally,

{p + (𝜈) | p ∈ ̃ 𝒫 and p𝐿-1 = 𝜇} 𝜇∈𝑁 𝐿-1 = {𝑝 ∈ 𝒫 | 𝑝 𝐿 = 𝜈} and similarly, {q + (𝜈) | q ∈ ̃ 𝒬 ℓ and q𝐿-1 = 𝜇} 𝜇∈𝑁 𝐿-1 = {𝑞 ∈ 𝒬 ℓ→𝐿 | 𝑞 𝐿 = 𝜈}. Thus, 𝑅 𝜃 (𝑥) 𝜈 = ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) + 𝐿-1 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ→𝐿 𝑞 𝐿 =𝜈 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥) + 𝑏 𝜈 = ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) + 𝐿 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ→𝐿 𝑞 𝐿 =𝜈 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥)
Proof of Proposition 3.3.2

Proposition. Let 𝒰(𝑥, 𝜃) be the set of all neighborhoods 𝑈 ⊂ R 𝑁 0 × R 𝐸∪𝐻 of (𝑥, 𝜃). We define, for any parameterization 𝜃,

𝒳 𝜃 {𝑥 ∈ R 𝑁 0 | ∃𝑈 ∈ 𝒰(𝑥, 𝜃), ∀𝑞 ∈ 𝒫∪𝒬, ∀(𝑥 ′ , 𝜃 ′ ) ∈ 𝑈, 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃 ′ , 𝑥 ′ )}. (A.
= (S𝛼) 𝜈 -(S𝛼) 𝜇 = - ∑ 𝑒 ′ ∈𝑞 𝛼 𝑒 ′ + ∑ 𝑒 ′ ∈𝑞 𝛼 𝑒 ′ = 𝛼 𝑒 .
S is surjective. Let 𝛾 ∈ R 𝐻 and 𝛼 ∈ R 𝐸 be defined for any edge 𝑒 as in (A.90).

Let 𝑞 = (𝑞 0 , . . . , 𝑞 𝐿 ) be a partial path going from some neuron

𝜇 = 𝑞 0 ∈ 𝑁 0 ∪ 𝐻 to the output neuron 𝜂 = 𝑞 𝑘 ∈ 𝑁 ℓ . If 𝜇 ∈ 𝑁 0 then ∑ 𝑒∈𝑞 𝛼 𝑒 = -𝛾 𝑞 𝑘-1 + 𝑘-1 ∑ 𝑗=2 (𝛾 𝑞 𝑗 -𝛾 𝑞 𝑗-1 ) + 𝛾 𝑞 1 = 0 hence 𝛼 ∈ ker P. If 𝜈 ∈ 𝐻 then similarly ∑ 𝑒∈𝑞 𝛼 𝑒 = -𝛾 𝑞 𝑘-1 + 𝑘-1 ∑ 𝑗=1 (𝛾 𝑞 𝑗 -𝛾 𝑞 𝑗-1 ) = 𝛾 𝑞 0 = 𝛾 𝜇 hence (S𝛼) 𝜇 = 𝛾 𝜇 . This shows that (S𝛼) = 𝛾. Proof of Proposition A.2.2 Proposition A.2.2. Consider 𝜃 = (𝑤, 𝑏) ∈ R 𝐸 × R 𝐻 a parameterization and define 𝑉 𝜃 Supp 𝑤 ×S -1 (Supp 𝑏 ) ⊂ R 𝐸 × ker P ⊂ R 𝐸 × R 𝐸 .
For any (𝛼, 𝛽) ∈ 𝑉 𝜃 , we write 𝛾 = (𝛼, S𝛽) ∈ Supp 𝑤 × Supp 𝑏 = Supp 𝜃 and

𝜙 𝜃 (𝛼, 𝛽) 𝜃 ⊙ 𝑒 𝛾 = (𝑤 ⊙ exp(𝛼), 𝑏 ⊙ exp(S𝛽))
where ⊙ denotes pointwise multiplication.

⎧ ⎪ ⎨ ⎪ ⎩ log ( 𝑤 ′ 𝑒 𝑤𝑒 ) if 𝑤 𝑒 ̸ = 0 0 if 𝑤 𝑒 = 0. (S𝛽) 𝜈 ⎧ ⎪ ⎨ ⎪ ⎩ log ( 𝑏 ′ 𝜈 𝑏𝜈 ) if 𝑏 𝜈 ̸ = 0 0 if 𝑏 𝜈 = 0. (A.
= α𝑒 = 0; if 𝑤 𝑒 ̸ = 0 then 𝑤 𝑒 exp (𝛼 𝑒 ) = 𝜙 𝜃 (𝛼, 𝛽) 𝑒 = 𝜙 𝜃 (α, β) 𝑒 = 𝑤 𝑒 exp (α 𝑒 ) implies 𝛼 𝑒 = α𝑒 . Overall we obtain 𝛼 = α.
Consider now 𝜈 ∈ 𝐻. As 𝛽, β ∈ S -1 (Supp 𝑏 ), we have S𝛽, S β ∈ Supp 𝑏 . When 𝑏 𝜈 = 0 this implies (S𝛽) 𝜈 = 0 = (S β) 𝜈 ; when 𝑏 𝜈 ̸ = 0, as 𝑏 𝜈 exp((S𝛽) 𝜈 ) = 𝜙 𝜃 (𝛼, 𝛽) 𝜈 = 𝜙 𝜃 (α, β) 𝜈 = 𝑏 𝜈 exp((S β) 𝜈 ) we get (S𝛽) 𝜈 = (S β) 𝜈 ; in both cases we obtain S𝛽 = S β. Since S is an isomorphism, we deduce that 𝛽 = β.

𝜙 𝜃 is surjective. Let 𝜃 ′ = (𝑤 ′ , 𝑏 ′ ) ∈ Sign 𝜃 . Let us show that Equations (A.91) define (𝛼, 𝛽 = S -1 (𝛾)) ∈ 𝑉 𝜃 such that 𝜃 ′ = 𝜙 𝜃 (𝛼, 𝛽).

Since 𝜃 ′ ∈ Sign 𝜃 , 𝑤 ′ 𝑒 and 𝑤 𝑒 have the same sign for every edge 𝑒 ∈ 𝐸. In particular, when 𝑤 𝑒 ̸ = 0 we get 𝑤 ′ 𝑒 /𝑤 𝑒 > 0, hence 𝛼 𝑒 is well defined, and 𝛼 ∈ Supp 𝑤 .

Similarly, 𝛾 𝜈 is well-defined for every hidden neuron 𝜈 ∈ 𝐻, and 𝛾 ∈ Supp 𝑏 , hence 𝛽 = S -1 𝛾 satisfies 𝛽 ∈ S -1 (Supp 𝑏 ). We observe that 𝜃 ′ = 𝜙 𝜃 (𝛼, 𝛽) to conclude.

Proof of Proposition 3.3.7

Proposition. Consider 𝜃 a parametrization and 𝐵 𝜃 {𝛾 ∈ Supp 𝜃 | ‖𝛾‖ 2 ≤ 1}. Then, for each 𝑥 ∈ 𝒳 𝜃 , there exists 𝜀 > 0 such that, for every (𝛾, 𝑡) ∈ 𝐵 𝜃 ×] -𝜀, 𝜀[, the following result holds:

𝑅 𝜃 (𝜃 ⊙ 𝑒 𝛾𝑡 , 𝑥) = ∑ 𝑝∈𝒫 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑒 𝑡P(𝛼)𝑝 𝑎 𝑝 (𝜃, 𝑥) + ∑ 𝑞∈𝒬 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑒 𝑡Q(𝛼-𝛽)𝑞 𝑎 𝑞 (𝜃, 𝑥). (A.92)
Proof. The proof follows two steps.

Neighborhood. Let 𝑥 ∈ 𝑋 𝜃 . Let us show that there exists a neighborhood Ω of 𝑡 = 0 such that, for every full or partial path 𝑞 ∈ 𝒫 ∪𝒬 and for every (𝛼, 𝛽, 𝑡) ∈ 𝐵 𝜃 ×Ω, we have 𝑎 𝑞 (𝜃 ⊙ 𝑒 𝛾𝑡 , 𝑥) = 𝑎 𝑞 (𝜃, 𝑥).

Let 𝑞 ∈ 𝒫 ∪ 𝒬 be a full path or partial path. First we show that there exists a neighborhood Ω 𝑞 of 𝑡 = 0 such that, for every (𝛼, 𝛽, 𝑡) ∈ 𝐵 𝜃 × Ω 𝑞 , 𝑎 𝑞 (𝜃 ⊙ 𝑒 𝛾𝑡 , 𝑥) = 𝑎 𝑞 (𝜃, 𝑥). We define the following continuous function:

𝑓 𝜃 : 𝑉 𝜃 × R -→ R 𝐸∪𝐻 (𝛼, 𝛽, 𝑡) ↦ → 𝜃 ⊙ 𝑒 𝛾𝑡 . Denote, for any set 𝑍 ∈ R 𝐸∪𝐻 , 𝑓 -1 𝜃 (𝑍) = {(𝛼, 𝛽, 𝑡) ∈ 𝑉 𝜃 × R | 𝑓 𝜃 (𝛼, 𝛽, 𝑡) ∈ 𝑍}. By Proposition 3.3.2, there exists an open neighborhood 4 𝑍 ⊂ R 𝐸∪𝐻 of 𝜃 such that, for all 𝜃 ′ ∈ 𝑍, 𝑎 𝑞 (𝜃 ′ , 𝑥) = 𝑎 𝑞 (𝜃, 𝑥). Define 𝑈 𝑓 -1 𝜃 (𝑍 ∩ Sign 𝜃 ) ⊂ 𝑉 𝜃 × R. (A.93) Since 𝑓 𝜃 is continuous and since 𝑍 is an open set in Sign 𝜃 , 𝑈 is an open set in 𝑉 𝜃 × R. Moreover, 𝑈 is a neighborhood of (0, 0, 0) ∈ 𝑉 𝜃 × R since 𝑓 𝜃 (0, 0, 0) = 𝜃 ∈ 𝑍 ∩ Sign 𝜃 . Define 𝐵 𝜃 (𝜀) {𝛾 ∈ Supp 𝜃 | ‖𝛾‖ 2 ≤ 𝜀}.
𝑓 𝜃 (𝐵 𝜃 (1) × Ω 𝑞 ) = 𝑓 𝜃 (𝐵 𝜃 (𝜀 𝑞 ) × ̃ Ω 𝑞 ) ⊂ 𝑓 𝜃 (𝑈 ) = 𝑍 ∩ Sign 𝜃 . (A.95)
Thus, for every (𝛼, 𝛽, 𝑡) ∈ 𝐵 𝜃 × Ω 𝑞 , we have 𝜃 ⊙ 𝑒 𝛾𝑡 ∈ 𝑍 ∩ Sign 𝜃 and therefore 𝑎 𝑞 (𝜃 ⊙ 𝑒 𝛾 , 𝑥) = 𝑎 𝑞 (𝜃, 𝑥). We finally define Ω = ∩ 𝑞 Ω 𝑞 . Since there are finitely many partial paths, Ω is also a neighborhood of 𝑡 = 0.

Computation. Let 𝑥 ∈ 𝑋 𝜃 and (𝛼, 𝛽, 𝑡) ∈ Ω. We note 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾𝑡 . For any path or partial path 𝑞 ∈ 𝒫 ∪ 𝒬, the following holds:

• Using the first proof step, 𝑎 𝑞 (𝜃 ′ , 𝑥) = 𝑎 𝑞 (𝜃, 𝑥).

• The expression of 𝑣 𝑞 (𝜃 ′ ) yields 𝑣 𝑞 (𝜃 ′ ) = 𝑣 𝑞 (𝜃) exp(𝑡 ∑ 𝑒∈𝑞 𝛼 𝑒 ).

• By definition of S (Proposition 3.3.6), the biases associated with the parameterization 𝜃 ′ are equal to 𝑏 ′ 𝜈 = 𝑏 𝜈 exp(𝑡S(𝛽) 𝜈 ) for every hidden neuron 𝜈 ∈ 𝐻.

We explicitly compute 𝜃 ⊙ 𝑒 𝛾𝑡 as follows. Using Proposition 3.3.1, we have

𝜃 ⊙ 𝑒 𝛾𝑡 = ∑ 𝑝∈𝒫 𝑥 𝑝 0 𝑣 𝑝 (𝜃 ′ )𝑎 𝑝 (𝜃 ′ , 𝑥) ⏟ ⏞ 𝜌 1 (𝑡) + ∑ 𝑞∈𝒬 𝑏 ′ 𝑞 0 𝑣 𝑞 (𝜃 ′ )𝑎 𝑞 (𝜃 ′ , 𝑥) ⏟ ⏞ 𝜌 2 (𝑡)
.

Using the above observations and Definition 3.3.1, the first term writes

𝜌 1 (𝑡) = ∑ 𝑝∈𝒫 𝑥 𝑝 0 𝑣 𝑝 (𝜃) exp ⎛ ⎝ 𝑡 ∑ 𝑒∈𝑝 𝛼 𝑒 ⎞ ⎠ 𝑎 𝑝 (𝜃, 𝑥) = ∑ 𝑝∈𝒫 𝑥 𝑝 0 𝑣 𝑝 (𝜃) exp (𝑡(P𝛼) 𝑝 ) 𝑎 𝑝 (𝜃, 𝑥).
Similarly, the second term writes

𝜌 2 (𝑡) = ∑ 𝑞∈𝒬 𝑏 𝑞 0 exp (𝑡S(𝛽) 𝑞 0 ) 𝑣 𝑞 (𝜃) exp ⎛ ⎝ 𝑡 ∑ 𝑒∈𝑞 𝛼 𝑒 ⎞ ⎠ 𝑎 𝑞 (𝜃, 𝑥).
In particular, as the definition of (S𝛽) 𝑞 0 is independent of the choice of the partial path going from 𝑞 0 to the output neuron (see Proposition 3.3.6), we have

𝜌 2 (𝑡) = ∑ 𝑞∈𝒬 𝑏 𝑞 0 exp ⎛ ⎝ -𝑡 ∑ 𝑒∈𝑞 𝛽 𝑒 ⎞ ⎠ 𝑣 𝑞 (𝜃) exp ⎛ ⎝ 𝑡 ∑ 𝑒∈𝑞 𝛼 𝑒 ⎞ ⎠ 𝑎 𝑞 (𝜃, 𝑥) = ∑ 𝑞∈𝒬 𝑏 𝑞 0 exp (-𝑡Q(𝛽) 𝑞 ) 𝑣 𝑞 (𝜃) exp (𝑡Q(𝛼) 𝑞 ) 𝑎 𝑞 (𝜃, 𝑥) = ∑ 𝑞∈𝒬 𝑏 𝑞 0 𝑣 𝑞 (𝜃) exp (𝑡Q(𝛼 -𝛽) 𝑞 ) 𝑎 𝑞 (𝜃, 𝑥).

A.2.3 Algebraic Characterization of Rescaling Equivalence

Proof of Proposition 3.3.8

Proposition. Sincomp 𝜃 is compact and Sincomp 𝜃 ⊂ Comp 𝜃 for any admissible parameterization 𝜃.

Proof. We prove both results successively.

Step 1: projection. We write (𝛼, 𝛽) = (𝛼 ⊥ , 𝛽 ⊥ ) + (𝛼 ‖ , 𝛽 ‖ ) where (𝛼 ⊥ , 𝛽 ⊥ ) ∈ 𝑉 𝜃 is non 𝜃-rescaling-compatible, 𝛼 ⊥ ∈ (ker (P 𝜃 )) ⊥ , and

𝑅 𝜙 𝜃 (𝛼 ⊥ ,𝛽 ⊥ ) = 𝑅 𝜙 𝜃 (𝛼,𝛽) = 𝑅 𝜃 . (A.122) First, decompose 𝛼 ∈ Supp 𝑤 ⊂ R 𝐸 as 𝛼 = 𝛼 ⊥ + 𝛼 ‖ , where ⎧ ⎪ ⎨ ⎪ ⎩ 𝛼 ⊥ ∈ (ker (P 𝜃 )) ⊥ 𝛼 ‖ ∈ ker (P 𝜃 )
We next construct 𝛽 ‖ as follows. We now extend the definition of S : : ker P → R 𝐻 to T : ker P 𝜃 → R 𝐻 (see Proposition 3.3.6) and similarly prove that T is well-defined (but T is not an isomorphism). Next, we define

𝛿 = T(𝛼 ‖ ) ∈ R 𝐻 . Next, define  𝛿 𝜈 ⎧ ⎨ ⎩ 𝛿 𝜈 if 𝑏 𝜈 ̸ = 0 0 if 𝑏 𝜈 = 0 (A.123)
Finally, define 𝛽 ‖ S -1 ( ̃ 𝛿) ∈ ker P and 𝛽 ⊥ 𝛽-𝛽 ‖ . We now show that (𝛼 ⊥ , 𝛽 ⊥ ) ∈ 𝑉 𝜃 is non 𝜃-rescaling-compatible.

• Let us show that 𝛼 ‖ , 𝛼 ⊥ ∈ Supp 𝑤 . First, by definition,

𝛼 ⊥ ∈ (ker (P 𝜃 )) ⊥ = Im ( P 𝑇 𝐷 𝒫 (𝜃) )
, where 𝐷 𝒫 (𝜃) is a diagonal matrix such that 𝐷 𝒫 (𝜃) 𝑝,𝑝 = 1(𝑣 𝑝 (𝜃) ̸ = 0) Given 𝑒 ∈ 𝐸 such that 𝑤 𝑒 = 0, we show that 𝛼 ⊥ 𝑒 = 0. It suffices to show that for every path 𝑝 ∈ 𝒫, hence 𝛼 ⊥ 𝑒 = 0. This shows that 𝛼 ⊥ ∈ Supp 𝑤 . As 𝛼 ∈ Supp 𝑤 , it follows that 𝛼 ‖ = 𝛼 -𝛼 ⊥ ∈ Supp 𝑤 .

• Let us show that 𝛽 ‖ , 𝛽 ⊥ ∈ S -1 (Supp 𝑏 ). Using Equation (A.123), we deduce that ̃ 𝛿 ∈ Supp 𝑏 . Hence, 𝛽 ‖ ∈ S -1 (Supp 𝑏 ). Since 𝛽 ∈ S -1 (Supp 𝑏 ), we also have 𝛽 ⊥ = 𝛽 -𝛽 ‖ ∈ S -1 (Supp 𝑏 ).

• Let us show that 𝑤 ‖ -𝛽 ‖ ∈ ker (Q 𝜃 ). It suffices to show that, for every partial path 𝑞 ∈ 𝒬, We conclude using the fact that 𝛿 𝑞 0 = ̃ 𝛿 𝑞 0 .

𝑏
As 𝛼 ‖ ∈ ker P 𝜃 and 𝛼 ‖ -𝛽 ‖ ∈ ker Q 𝜃 , we get that (𝛼 ‖ , 𝛽 ‖ ) is 𝜃-rescaling compatible. Since (𝛼, 𝛽) = (𝛼 ⊥ , 𝛽 ⊥ ) + (𝛼 ‖ , 𝛽 ‖ ) is non 𝜃-rescaling compatible, this implies that (𝛼 ⊥ , 𝛽 ⊥ ) is non 𝜃-rescaling compatible as claimed. Using the above remarks, let us finally show that 𝑅 𝜙 𝜃 (𝛼,𝛽) = 𝑅 𝜙 𝜃 (𝛼 ⊥ ,𝛽 ⊥ ) (A.128)

Define 𝜃 0 = 𝜙 𝜃 (𝛼 ⊥ , 𝛽 ⊥ ). Let us show that (𝛼 ‖ , 𝛽 ‖ ) is 𝜃 0 -rescaling-compatible. Since (𝛼 ‖ , 𝛽 ‖ ) is 𝜃-rescaling-compatible, if suffices to show that 𝐷 𝒫 (𝜃) = 𝐷 𝒫 (𝜃 0 ) and 𝐷 𝒬 (𝜃) = 𝐷 𝒬 (𝜃 0 ). This is indeed a trivial consequence of the fact that sign(𝜃) = sign(𝜃 0 ), hence (𝛼 ‖ , 𝛽 ‖ ) is 𝜃 0 -rescaling-compatible. Next, using Proposition 3.3.2, we deduce that

𝑅 𝜙 𝜃 0 (𝛼 ‖ ,𝛽 ‖ ) = 𝑅 𝜃 0 (A.129)
We conclude by using the fact that 𝜙 𝜃 0 (𝛼 ‖ , 𝛽 ‖ ) = 𝜙 𝜃 (𝛼, 𝛽).

Step 2: normalization. Set 𝜏 ‖(𝛼 ⊥ , 𝛽 ⊥ )‖ 2 . As (𝛼 ⊥ , 𝛽 ⊥ ) is non 𝜃-rescalingcompatible, it is nonzero hence 𝜏 > 0. We can thus define where 𝑧 is a quantity that does not depend on the choice of 𝑥 + and 𝑥 -defined as

𝑧 = ∑ 𝜈 ′ ∈𝑁 1 𝜈 ′ ̸ =𝜈 𝑎 𝜈 ′ (𝑥 + )𝑑 𝜈 ′ . (A.160)
Let 𝜏 ∈ R 𝑁 0 be any direction such that 𝑥 + + 𝜏 ∈ Ω ∪ Γ + 𝜈 and 𝑥 -+ 𝜏 ∈ Ω ∪ Γ - 𝜈 . Evaluating Equation (A.159) at 𝑥 + + 𝜏 and 𝑥 -+ 𝜏 yields ⟨𝑥 --𝑥 + , 𝑧⟩ -⟨𝑑 𝜈 , 𝑥 + ⟩ -𝑒 𝜈 = ⟨𝑑 𝜈 , 𝜏 ⟩ .

(A.161)

Since the left-hand side of Equation (A.159) does not depend on 𝜏 , we deduce that 𝑑 𝜈 = 0. Since this is valid for every 𝜈 ′ ∈ 𝑁 1 , we deduce from Equation (A.160) that 𝑧 = 0. Then, Equation (A.159) implies that 𝑒 𝜈 = 0. We apply the same reasoning to every 𝜈 ∈ 𝐻 to deduce that 𝒞 ⊥ 𝜃 = {0}.

Proof of Proposition 3.4.8

Proposition. Let 𝜃 be an admissible parameterization such that 𝒞 ⊥ 𝜃 = {0}. Then, 𝜃 is locally identifiable.

Proof. Let us show that 𝜃 is locally identifiable. For the sake of contradiction, let us assume that 𝜃 is locally non-identifiable. Then, using Proposition 3.4.7, we must have both 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 = {0} and 𝑢(𝜃) ̸ = 𝑢(𝜃 ′ ), which is a contradiction.

B.1.1 Gradients & Biases

Denoting the rescaled weights and biases with a tilde, and denoting by ℒ the loss of the network, for every layer ℓ ∈ 0, 𝐿 , we have 𝜕ℒ 𝜕 ̃ 𝑦 (ℓ) = 𝜕ℒ 𝜕𝑦 (ℓ) ( 𝐷 (ℓ) ) -1 .

(B.1)

Similarly, we obtain 𝜕ℒ 𝜕 ̃ 𝑊 (ℓ) = 𝐷 (ℓ-1) 𝜕ℒ 𝜕𝑊 (ℓ) (

𝐷 ℓ

) -1 and 𝜕ℒ 𝜕 ̃ 𝑏 (ℓ) = 𝜕ℒ 𝜕𝑏 (ℓ) ( 𝐷 (ℓ) ) -1 .

(B.2) Equation (B.2) will be used to update the momentum (see Section 4.5).

B.2 Proof of Convergence of Equi-normalization

We now prove Theorem 4.3.1. We use the framework of block coordinate descent and we first state a consequence of a theorem of [START_REF] Tseng | Convergence of a block coordinate descent method for nondifferentiable minimization[END_REF] )}

. We make the following assumptions:

(1) 𝑓 is differentiable on 𝐷 ;

(2) 𝑋 is compact ;

(3) for each 𝑥 ∈ 𝐷, each block coordinate function 𝑓 ℓ : 𝑡 → 𝑓 (𝑥 1 , . . . , 𝑥 ℓ-1 , 𝑡, 𝑥 ℓ+1 , . . . , 𝑥 𝐵 ),

where 2 ≤ ℓ ≤ 𝐵 -1, has at most one minimum.

Under these assumptions, the sequence (𝑥 (𝑟) ) 𝑟∈N generated by the coordinate descent algorithm is defined and bounded. Moreover, every cluster point of (𝑥 (𝑟) ) 𝑟∈N is a local minimizer of 𝑓 .

Step 1. We apply Theorem B.2.1 to the function 𝜙. This is possible because all the assumptions are verified as shown below. Recall that

𝜙(𝑑) = 𝐿 ∑ ℓ=1 ⃦ ⃦ ⃦ ⃦ ( 𝐷 (ℓ-1) ) -1 𝑊 (ℓ) 𝐷 (ℓ) ⃦ ⃦ ⃦ ⃦ 𝑝 𝑝 = 𝐿 ∑ ℓ=1 ∑ 𝑖,𝑗 ⃒ ⃒ ⃒ ⃒ ⃒ 𝑑 ℓ [𝑗] 𝑑 ℓ-1 [𝑖]
𝑊 (ℓ) [𝑖, 𝑗] • For the first hidden layer, index ℓ = 1. By assumption, every hidden neuron is connected at least to one neuron in the input layer. Thus, for every 𝑗, there exists 𝑖 such that 𝑊 (1) [𝑖, 𝑗] ̸ = 0. Because 𝜙(𝑑) < 𝑀 𝑝 and 𝑑 0 [𝑖] = 1 for all 𝑖,

(𝑑 1 [𝑗]) 𝑝 ⃒ ⃒ ⃒𝑊 (1) [𝑖, 𝑗] ⃒ ⃒ ⃒ 𝑝 = ( 𝑑 1 [𝑗] 𝑑 0 [𝑖] ) 𝑝 ⃒ ⃒ ⃒𝑊 (1) [𝑖, 𝑗] ⃒ ⃒ ⃒ 𝑝 < 𝑀 𝑝 (B.5)
Thus ‖𝑑 1 ‖ ∞ < 𝐶𝑀 .

• For any next hidden layer, index ℓ ∈ 2, 𝐿 -1 . By assumption, every hidden neuron is connected at least to one neuron in the previous layer. Thus, for every 𝑗, there exists 𝑖 such that 𝑊 (ℓ) [𝑖, 𝑗] ̸ = 0. Because 𝜙(𝑑) < 𝑀 , Using the induction hypothesis, we get ‖𝑑 𝑘 ‖ ∞ < (𝐶𝑀 ) ℓ .

Thus, ‖𝑑‖ ∞ < (𝑀 𝐶) 𝐿 because 𝑀 𝐶 > 1. By contraposition, 𝜙 → +∞.

Thus, there exists a ball 𝐵 such that 𝑑 / ∈ 𝐵 implies 𝜙(𝑑) > 𝜙(𝑑 (0) ). Thus, by contraposition, 𝑑 ∈ 𝑋 implies that 𝑥 ∈ 𝐵. Thus, 𝑋 ⊂ 𝐵 is bounded. Moreover, 𝑋 is closed because 𝜙 is continuous thus 𝑋 is compact. Then, Assumption (2) is satisfied. 4.2). The existence and uniqueness of the minimum comes from the fact that each hidden neuron is connected to at least one input and one output neuron, thus all the row and column norms of the hidden weight matrices 𝑊 (ℓ) are non-zero, as well as the column (resp. row) norms or 𝑊 (1) (resp. 𝑊 (𝐿) ).

Assumption (3). We next note that

Step 2. We prove that 𝜙 has at most one stationary point on 𝐷 under the assumption that each hidden neuron is connected either to an input neuron or to an output neuron, which is weaker than the general admissibility assumption of Theorem 4.3.1 that assumes that each hidden neuron is connected to an input neuron and to an output neuron.

As defined in the Notations, we denote the set of all neurons in the network by 𝑉 , the set of all hidden neurons by 𝐻. For each neuron 𝜈, we define prev(𝜈) as the neurons connected to 𝜈 that belong to the previous layer. We further denote by 𝐻 the set of hidden neurons 𝜈 belonging to layers ℓ ∈ 1, 𝐿 -1 . We define 𝐸 0 as the set of edges whose weights are non-zero, i.e. We now show that 𝜙 has at most one stationary point on 𝐷. Directly computing the gradient of 𝜙 and solving for zeros happens to be painful or even intractable. Thus, we define a change of variables as follows. We define ℎ as ℎ : (0, +∞) 𝐻 → R 𝐻 𝑑 ↦ → log(𝑑)

• 𝑘 = 1. Let 𝜈 be a neuron at layer 1. Then, there exists a path coming from an input neuron to 𝜈 0 through edge 𝑒 1 . By definition, 𝑃 (𝑥) 𝜈 0 = 0 and 𝑃 (𝑥) 𝜈 = 𝑥 𝜈 , hence 𝑆 𝐻 (𝑥) 𝑒 1 = 𝑥 𝜈 -0. Since 𝑆 𝐻 (𝑥) = 0 it follows that 𝑥 𝜈 = 0.

• 𝑘 → 𝑘 + 1. Same reasoning using the fact that 𝑥 𝜈 𝑘 = 0 by induction.

The case where the path goes from neuron 𝜈 to some output neuron is similar.

B.3 Extension of ENorm to CNNs

B.3.1 Convolutional Layers

Let us consider two consecutive convolutional layers ℓ and ℓ + 1, without bias. Layer ℓ has 𝐶 ℓ filters of size 𝐶 ℓ-1 × 𝑆 ℓ × 𝑆 ℓ , where 𝐶 ℓ-1 is the number of input features and 𝑆 ℓ is the kernel size. This results in a weight tensor 𝑇 ℓ of size 𝐶 ℓ × 𝐶 ℓ-1 × 𝑆 ℓ × 𝑆 ℓ . Similarly, layer ℓ + 1 has a weight matrix 𝑇 ℓ+1 of size 𝐶 ℓ+1 × 𝐶 ℓ × 𝑆 ℓ+1 × 𝑆 ℓ+1 . We then perform axis-permutation and reshaping operations to obtain the 2D matrices:

𝑀 ℓ of size (𝐶 ℓ-1 × 𝑆 ℓ × 𝑆 ℓ ) × 𝐶 ℓ ;
𝑀 ℓ+1 of size 𝐶 ℓ × (𝐶 ℓ+1 × 𝑆 ℓ+1 × 𝑆 ℓ+1 ) .

(B.11) (B.12)

For example, we first reshape 𝑇 ℓ as a 2D matrix by collapsing its last 3 dimensions, then transpose it to obtain 𝑀 ℓ . We then jointly rescale these 2D matrices using rescaling matrices 𝐷 ℓ ∈ 𝒟(ℓ) as detailed in Section 4.3 and perform the inverse axis permutation and reshaping operations to obtain a right-rescaled weight tensor  𝑇 ℓ and a left-rescaled weight tensor ̃ 𝑇 ℓ+1 . See Figure 4-2 for an illustration of the procedure. This matched rescaling does preserve the function implemented by the composition of the two layers, whether they are interleaved with a ReLU or not. It can be applied to any two consecutive convolutional layers with various stride and padding parameters. When the kernel size is 1 in both layers, we recover the fullyconnected case of Figure 4-1.

B.3.2 Skip Connections

We now consider an elementary block of a ResNet-18 architecture as depicted in Figure 4-3. In order to maintain functional equivalence, we only consider ResNet architectures of type C as defined in (He et al., 2015a), where all shortcuts are learned 1 × 1 convolutions.

Structure of the rescaling process. Consider a ResNet block ℓ. We first leftrescale the Conv1 and ConvSkip weights using the rescaling coefficients calculated between blocks ℓ -1 and ℓ. We then rescale the two consecutive layers Conv1 and Conv2 with their own rescaling coefficients, and finally right-rescale the Conv2 and ConvSkip weights using the rescaling coefficients calculated between blocks ℓ and ℓ+1.

Computation of the rescaling coefficients. Two types of rescaling coefficients are involved, namely these between two convolution layers inside the same block and these between two blocks. The rescaling coefficients between the Conv1 and Conv2 layers are calculated as explained in Section 4.4.1. Then, in order to calculate the rescaling coefficients between two blocks, we compute equivalent block weights to deduce rescaling coefficients.

We empirically explored some methods to compute the equivalent weight of a block using electrical network analogies. The most accurate method we found is to compute the equivalent weight of the Conv1 and Conv2 layers, i.e., to express the succession of two convolution layers as only one convolution layer denoted as ConvEquiv (series equivalent weight), and in turn to express the two remaining parallel layers ConvEquiv and ConvSkip again as a single convolution layer (parallel equivalent weight). It is not possible to obtain series of equivalent weights, in particular when the convolution layers are interleaved with ReLUs. Therefore, we approximate the equivalent weight as the parallel equivalent weight of the Conv1 and ConvSkip layers.

B.4 Implicit Equi-normalization

In Section 4.3, we defined an iterative algorithm that minimizes the global ℓ 𝑝 norm of the network where 𝜆 is a regularization parameter. We now consider both the weights and the rescaling coefficients as learnable parameters and we rely on automatic differentiation packages to compute the derivatives of ℒ with respect to the weights and to the rescaling coefficients. We then train the network by performing iterative SGD steps and updating all the learnt parameters. By design, the derivative of 𝒞 with respect to any rescaling coefficient is zero. Although the additional overhead of implicit ENorm is theoretically negligible, we observed an increase of the training time of a ResNet-18 by roughly 30% using PyTorch 4.0 [START_REF] Paszke | Automatic differentiation in pytorch[END_REF]. We refer to Implicit Equi-normalization as ENorm-Impl and to Explicit Equi-normalization as ENorm.

We performed early experiments for the CIFAR10 fully-connected case. ENorm-Impl performs generally better than the baseline but does not outperform explicit ENorm, in particular when the network is deep. We follow the same experimental setup than previously, except that we additionally cross-validated 𝜆. We also initialize all the rescaling coefficients to one. Recall that ENorm or ENorm denotes explicit Equi-normalization while ENorm-Impl denotes Implicit Equi-normalization. We did not investigate learning the weights and the rescaling coefficients at different speeds (e.g. with different learning rates or momentum). This may explain in part why ENorm-Impl generally underperforms ENorm in these early experiments.

B.5 Experiments

We perform sanity checks to verify our implementation and give additional results.

B.5.1 Sanity Checks

We apply our Equi-normalization algorithm to a ResNet architecture by integrating all the methods exposed in Section 4.4. We perform three sanity checks before proceeding to experiments. First, we randomly initialize a ResNet-18 and verify that it outputs the same probabilities before and after balancing. Second, we randomly initialize a ResNet-18 and perform successive ENorm cycles (without any training) and observe that the 𝐿 2 norm of the weights in the network is decreasing and then converging, as theoretically proven in Section 4.3, see Figure B-3.

We finally compare the evolution of the total ℓ 2 norm of the network when training it, with or without ENorm. We use the setup described in Subsection 4.6. We report the influence of the proportion of blocks to which we apply the noise. We focus on Transformer for Wikitext-103 language modeling. We explore two settings: iPQ and int8. For iPQ, we use 𝜙 proxy .

C.2.2 Impact of Approximating the Noise Function

We study the impact of approximating quantization noise during training. We focus on the case of iPQ with the approximation described in Section 6.4.2. In Table C.2, we compare the correct noise function for iPQ with its approximation 𝜙 proxy . This approximate noise function does not consider cluster assignments or centroid values and simply zeroes out the selected blocks. For completeness, we include an intermediate approximation where we consider cluster assignments to apply noise within each cluster, but still zero-out the vectors. These approximations do not affect the performance of the quantized models. This suggests that increasing the correlation between subvectors that are jointly clustered is enough to maintain the performance of a model quantized with iPQ. Since PQ tends to work well on highly correlated vectors, such as activations in convolutional networks, this is not surprising. Using the approximation 𝜙 proxy presents the advantage of speed and practicality. Indeed, one does not need to compute cluster assignments and centroids for every layer in the network after each epoch. Moreover, this approach is less involved in terms of code.

C.5.4 Difficulty of Quantizing Different Model Structures

Quantization is applied to various portions of the Transformer architecture -the embedding, attention, feedforward, and classifier output. We compare the quantizability of various portions of the network in this section.

Is the order of structures important? We quantize specific network structures first -this is important as quantizing weight matrices can accumulate reconstruction error. Some structures of the network should be quantized last so the finetuning process can better adjust the centroids. We find that there are small variations in performance based on quantization order (see . We choose to quantize Which structures can be compressed the most? Finally, we analyze which network structures can be most compressed. During quantization, various matrix block sizes can be chosen as a parameter -the larger the block size, the more compression, but also the larger the potential reduction of performance. Thus, it is important to understand how much each network structure can be compressed to reduce the memory footprint of the final model as much as possible. In Figure C-4, we quantize two model structures with a fixed block size and vary the block size of the third between 4 and 32. As shown, the FFN and embedding structures are more robust to aggressive compression, while the attention drastically loses performance as larger block sizes are used.

C.5.5 Approach to intN Scalar Quantization

We compare quantizing per-channel to using a histogram quantizer in Table C.7.

The histogram quantizer maintains a running min/max and minimizes L2 distance between quantized and non-quantized values to find the optimal min/max. Quantizing per channel learns scales and offsets as vectors along the channel dimension, which provides more flexibility since scales and offsets can be different.

C.5.6 LayerDrop with STE

For quantization noise, we apply the straight through estimator (STE) to remaining weights in the backward pass. We experiment with applying STE to the backward We compare in Table D.2 different variants of the Motion Net approach, using 20 input landmarks, 68 input landmarks, self-supervised landmarks with dense architectures and with mobile architectures. All these dense architecture employ a latent space of 256 × 64 × 64, and were trained on VoxCeleb. We first note that using standard facial landmarks instead of unsupervised motion landmarks degrades the scores of all metrics. Using 68 landmarks is not improving the quality over 20. With mobile architectures, we reduce the latent space to 256 × 32 × 32. In addition to using 10 motion landmarks or 20 landmarks alone, combining these two sets helps boost all quality metrics. Finally, we observe that adding the SPADE blocks preserves the perceptual quality and brings a large improvement in NME.

D.1.2 Quantitative Comparative Evaluation

We compare the quality/bandwidth trade-off of different dense face animation approaches in Table D.3. As the different models were trained using different data pre-processing (different crops, alignment), we evaluate each one in the setting allowing the best generations. This means that synthesized videos need to be compared to different source videos. Therefore, we paste the ground truth video background on the generated result so that the metrics focus on evaluating face differences only. We observe that the NTH results lead to better NME, and FOM to better LPIPS and CSIM metrics. SegFace numerical results lower, partly due to a color shift appearing in the results. Interestingly, our mobile results have better NME and msVGG scores than the original FOM dense approach.

D

  .1.1 Quality Evaluation: Ablation Studies . . . . . . . . . . . . . . 221 D.1.2 Quantitative Comparative Evaluation . . . . . . . . . . . . . . 223

Figure 2

 2 Figure 2-1: The double-descent phenomenon in Deep Learning as illustrated by Nakkiran et al. (2019). Here the complexity of the considered model (a ResNet-18) is measured as its width parameter, where higher width denotes a network with a larger number of parameters.

Figure 2

 2 Figure 2-4: Illustration of the rescaling operations on one neuron, 𝜈. Left: original network. Right: after applying the rescaling with 𝜆 > 0. For ReLU non-linearities, the function implemented by the network remains the same.

Figure 2 - 5 :

 25 Figure 2-5: Illustration of the permutation operations on two neurons, 𝜈 1 and 𝜈 2 . Right: original network. Left: after applying the permutation. The function implemented by the network remains the same for any activation function, and in particular for ReLU non-linearities.

Figure 2

 2 Figure 2-6: Left: A two-hidden layer network valued with a parameterization 𝜃.Weights are indicated on the connections between neurons: for instance, 𝑤 𝜇 1 →𝜈 1 = 0. Biases are specified next to the neurons: for instance 𝑏 𝜅 = 0. Input and output neurons are depicted as squares whereas hidden neurons are displayed as circles. Note that the input space is two-dimensional: 𝑥 = (𝑥 𝜇 1 , 𝑥 𝜇 2 ) and the output is scalar: |𝑁 𝐿 | = 1. Right: 3D visualization of the network's realization 𝑥 ↦ → 𝑅 𝜃 (𝑥) which is piecewise affine.

Figure 2

 2 Figure 2-7: Activation and linear regions for the two-hidden-layer network defined in Figure 2-6. Recall that the input 𝑥 = (𝑥 𝜇 1 , 𝑥 𝜇 2 ) is two-dimensional, i.e. |𝑁 0 | = 2 and the output is scalar, i.e. |𝑁 3 | = 1. Left: activation regions labeled with the activation pattern 𝜌(𝑥) for neurons (𝜈 1 , 𝜈 2 , 𝜂) given the input 𝑥. For instance, 𝜌(𝑥) = 011 means that the neuron 𝜈 1 is off whereas both neurons 𝜈 2 and 𝜂 are on for input 𝑥. The two blue lines represent the separating hyperplanes for neurons 𝜈 1 and 𝜈 2 whereas the orange line denotes the additional gradient discontinuities introduced by neuron 𝜂. Right: linear regions implemented by 𝑅 𝜃 . There are 7 activation regions and 4 linear regions. The figures are generated with our own PyTorch script available at https: //github.com/pierrestock/linear-regions/blob/main/partition.ipynb.

Figure 2

 2 Figure 2-8: Activation and linear regions for a three hidden layer neural network with |𝑁 1 | = 5 neurons on the first hidden layer and |𝑁 2 | = 4 neurons on the second hidden layer and |𝑁 3 | = 3 neurons on the third hidden layer. The input 𝑥 is twodimensional, i.e. |𝑁 0 | = 2 and the output is scalar, i.e. |𝑁 4 | = 1. Left: activation regions. Right: linear regions. All the weights and biases were initialized randomly. The figures are generated with our own PyTorch script available at https://github. com/pierrestock/linear-regions/blob/main/partition.ipynb.

Figure 2 -

 2 Figure 2-10: We compare a properly balanced network (blue) and an unbalanced network (red). Each dot represents the 𝐿 2 norm of one column in the layer's weight matrix. The dots are randomly spread around their layer index on the 𝑥-axis for readability. Both networks (balanced and unbalanced) implement the same inputoutput function.

34

  Figure 1(a) of Neyshabur et al. (2015). 35 See Chapter 3 for a formal derivation.

  He et al. (2017b); Huang et al. (2018b); Mittal et al. (2018); Luo et al. (2017); He et al. (2018a); Evci et al. (2019); Louizos et al. (2017b) for pruning variants.

Figure 2 -

 2 Figure 2-11: Accuracy for several levels of compression (left) and theoretical speedup (right) for a ResNet-18 on ImageNet reported by Blalock et al. (2020). Weight and Gradient refer respectively to weight magnitude pruning and gradient magnitude pruning. If these criteria are applied per layer, the method is prefixed with Layer. If the criteria are applied to the whole network, the method is prefixed with Global.

Figure 2 -

 2 Figure 2-12: ShuffleNet block (Zhang et al., 2017b). (a) no shuffle, (b) channel shuffle and (c) an equivalent implementation of (b).

Figure 2 -

 2 Figure 2-13: Tradeoffs in terms of FLOPs, number of parameters and total number of activations during a forward pass for EfficientNets and RegNets as exposed by Radosavovic et al. (2020). Error is test top-1 error on ImageNet object classification.

Figure 2 -

 2 Figure 2-14: Performing Product Quantization on a weight matrix 𝑊 .

Figure 2 -

 2 Figure 2-15: Left: Pareto curve of more than 250 image classification models on ImageNet with no added compression methods. Right: zoom on the bottom-left corner. Higher top-1 accuracy, lower latency and lower number of parameters is better. Timings were computed on a Quad-Core Intel Core i7, on a single thread.

Figure 3

 3 Figure 3-1: One hidden layer network architecture. Hidden neurons are depicted as circles whereas input and output neurons are displayed as squares.

Figure 3

 3 Figure3-2: Left: example of a dead neuron, 𝜈 2 . Intuitively, 𝜈 2 does not participate in the realization of the network 𝑅 𝜃 . Center: alternative parameterization 𝜃 ′ such that 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent, with 𝜀 > 0. Right: the same realization is implemented by 𝜃 and 𝜃 ′ . Squares denote input or output neurons whereas circles denote hidden neurons. Weights are specified on the connections between two neurons and all biases are set to zero (not represented here).

Figure 3

 3 Figure3-3: Left: example of siamese neurons, 𝜈 1 and 𝜈 2 . Intuitively, 𝜈 1 and 𝜈 2 fire in the same half-space. Center: alternative parameterization 𝜃 ′ such that 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent. Right: the same realization is implemented by 𝜃 and 𝜃 ′ , with 𝜀 > 0. Squares denote input or output neurons whereas circles denote hidden neurons. Weights are specified on the connections between two neurons and all biases are set to zero (not represented here).

Figure 3

 3 Figure3-4: Left: example of invisible neurons, 𝜈 1 and 𝜈 2 . Intuitively, 𝜈 1 and 𝜈 2 are such that their common breakpoint is not visible on the realization 𝑅 𝜃 . Center: alternative parameterization 𝜃 ′ such that 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent. Right: the same realization is implemented by 𝜃 and 𝜃 ′ , with 𝜀 > 0. Squares denote input or output neurons whereas circles denote hidden neurons. Weights are specified on the connections between two neurons and biases are explicitly specified.

Figure 3

 3 Figure 3-5: Left: example of cousin neurons, 𝜈 1 , 𝜈 2 and 𝜈 3 . Intuitively, 𝜈 1 , 𝜈 2 and 𝜈 3 are such that two of them can ether implement the leftmost or the rightmost breakpoint (see right). Center: alternative parameterization 𝜃 ′ such that 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent. Right: the same realization is implemented by 𝜃 and 𝜃 ′ , with 𝜀 > 0. Squares denote input or output neurons whereas circles denote hidden neurons. Weights are specified on the connections between two neurons and biases are explicitly specified.

  The function 𝑥 ↦ → 𝑅 𝜃 (𝑥) is piecewise affine continuous and locally affine in the neighborhood of any 𝑥 ∈ 𝒳 𝜃 .We state a lesser-known result about the piece-wise polynomial nature of the function 𝜃 ↦ → 𝑅 𝜃 (𝑥) which we prove in Appendix A.2.1. Proposition 3.3.4. For any 𝑥 ∈ 𝒳 𝜃 , the function 𝜃 ↦ → 𝑅 𝜃 (𝑥) is piecewise polynomial continuous and locally polynomial in the neighborhood of any 𝜃.

  Finally, Huang et al. (2017b) and[START_REF] Yuan | Scaling-based weight normalization for deep neural networks[END_REF] use a different projection technique to select the representant based on Riemannian gradients while Bernstein et al. (2020) derive a multiplicative version of the Adam optimizer.

Figure 4

 4 Figure 4-2: Rescaling the weights of two consecutive convolutional layers that preserves the function implemented by the CNN. Layer ℓ scales channel number 𝑖 of the input activations by 𝛾 𝑖 and layer ℓ + 1 cancels this scaling with the inverse scalar so that the activations after layer ℓ + 1 are unchanged.

Figure 4 - 3 :

 43 Figure 4-3: Rescaling an elementary block within a ResNet-18 consists of 3 steps. (1) Conv1 and ConvSkip are left-rescaled using the rescaling coefficients between blocks 𝑘 -1 and 𝑘; (2) Conv1 and Conv2 are rescaled as two usual convolutional layers; (3) Conv2 and ConvSkip are right-rescaled using the rescaling coefficients between blocks 𝑘 and 𝑘 + 1. Identical colors denote the same rescaling coefficients 𝐷. Coefficients between blocks are rescaled as detailed in Section B.3.2.

Figure 4

 4 Figure4-4: Left: MNIST auto-encoder results (lower is better). Right: CIFAR-10 fully-connected results (higher is better).

Figure 4

 4 Figure 4-5: ResNet-18 results on the ImageNet dataset, batch size 64.

Figure 5

 5 Figure 5-2: We quantize 𝐶 out filters of size 𝐶 in × 𝐾 × 𝐾 using a subvector size of 𝑑 = 𝐾 × 𝐾. In other words, we spatially quantize the convolutional filters to take advantage of the redundancy of information in the network. Similar colors denote subvectors assigned to the same codewords.

%%Figure 6 - 2 :

 62 Figure 6-2: Performance as a function of model size. We compare models quantized with PQ and trained with the related Quant-Noise to the state of the art. (a) Test perplexity on Wikitext-103 (b) Dev Accuracy on MNLI (c) ImageNet Top-1 accuracy. Model size is shown in megabytes on a log scale. Red and gray coloring indicates existing work, with different colors for visual distinction.

Figure 7

 7 Figure 7-4: Daniel (with glasses) is calling Maxime on our iPhone app. Left: Daniel's phone when calling Maxime. The reference frame of Daniel (top right) of Daniel is updated periodically. Daniel sees the face of Maxime reconstructed in real time. Right: on the generated frame, we display the set of landmarks used to perform the inference. The generation runs at 20 FPS on an iPhone 8 and the call uses an average video bandwidth of 5 kbits/s.

Proposition A.1. 1 .

 1 Two rescaling equivalent parameterizations 𝜃 and 𝜃 ′ yield the same realization, i.e. 𝑅 𝜃 (𝑥) = 𝑅 𝜃 ′ (𝑥) for all 𝑥. Proposition A.1.2. Two permutation equivalent parameterizations 𝜃 and 𝜃 ′ yield the same realization, i.e. 𝑅 𝜃 (𝑥) = 𝑅 𝜃 ′ (𝑥) for all 𝑥. Proposition A.1.3. Two parameter vectors 𝜃 and 𝜃 ′ are permutation-rescaling equivalent iff they are rescaling-permutation equivalent.

  We assume ∀𝑡, 𝜙(𝑡) = 𝜓(𝑡). Then, 𝑐 = 𝑐 ′ and up to a re-numbering, ℐ = ℐ ′ and for all 𝑖 ∈ ℐ, 𝑡 𝑖 = 𝑡 ′ 𝑖 , 𝑠 𝑖 = 𝑠 ′ 𝑖 and 𝑎 𝑖 = 𝑎 ′ 𝑖 .

  A.73) Let us denote 𝒥 = {𝑖 ∈ ℐ | 𝜀 𝑖 = -1} and let us show that 𝒥 = ∅. Using Equation (A.73), we re-write Equation (A.72) as follows:∑ 𝑖∈𝒥 𝑎 𝑖 𝑠 𝑖 (𝑡 -𝑡 𝑖 ) + 𝑐 -𝑐 ′ = 0 (A.74)Since this is valid for all 𝑡 ∈ R and since we assume that ∑ 𝑖∈𝒥 𝑎 𝑖 𝑠 𝑖 ̸ = 0, we obtain that 𝒥 = ∅, hence 𝑠 ′ 𝑖 = 𝑠 𝑖 for all 𝑖 ∈ ℐ. Finally, evaluating Equation (A.72) at any point 𝑡 yields 𝑐 = 𝑐 ′ , which concludes the proof. For any parameterization 𝜃 = (𝑤, 𝑏), any 𝑥 ∈ R 𝑁 0 and any output neuron 𝜈 ∈ 𝑁 𝐿 ,

  This is achieved by treating two cases:for 𝑝 such that 𝑒 / ∈ 𝑝, we have 𝑝,𝑒 = 1(𝑒 ∈ 𝑝) = 0;for 𝑝 such that 𝑒 ∈ 𝑝, since 𝑤 𝑒 = 0, 𝐷 𝒫 (𝜃) 𝑝,𝑝 = 1(𝑣 𝑝 (𝜃) ̸ = 0

  𝑞 0 𝑣 𝑞 (𝜃) to show (A.125) for any partial path 𝑞 such that 𝑏 𝑞 0 ̸ = 0 and 𝑣 𝑞 (𝜃) ̸ = 0. Let 𝑞 be such a partial path. Then,∑ 𝑒∈𝑞 𝛼 ‖ 𝑒 = -T(𝛼 ‖ ) 𝑞 0 = -𝛿 𝑞 0 (A.126)and∑ 𝑒∈𝑞 𝛽 ‖ 𝑒 = -S(𝛽 ‖ ) 𝑞 0 = -̃ 𝛿 𝑞 0 (A.127)

=

  𝑅 𝐺 (𝜃) = 𝜌 𝛾𝑛,𝑥 (0(𝜉 𝑛 ), hence, since 𝜏 𝑛 ̸ = 0, 𝜌 ′ 𝛾𝑛,𝑥 (0) = -𝜏 𝑛 2 𝜌 ′′ 𝛾𝑛,𝑥 (𝜉 𝑛 ).Using Equation (A.139), we deduce that, for 𝑛 large enough, (0)→ 𝑛→∞ 0. Since lim 𝑛→∞ 𝜌 ′ 𝛾𝑛,𝑥 (0) = 𝜌 ′ 𝛾,𝑥 (0), we deduce 𝜌 ′ 𝛾,𝑥 (0) = 0.Finally, differentiating the function 𝑡 ↦ → 𝜌 𝛾,𝑥 (𝑡) and evaluating in 𝑡 = 0 yields:𝜌 ′ 𝛾,𝑥 (0) = ⟨∇ 𝜃 𝑅 𝜃 (𝑥), 𝜃 ⊙ 𝛾⟩ (A.146)which concludes the proof.for all 𝑥 ∈ 𝒳 𝜃 . Let 𝜈 ∈ 𝑁 1 . Since there are no twin neurons, the separating hyperplanes Γ 𝜈 ′ defined in 3.2.2 are all distinct for 𝜈 ′ ∈ 𝑁 1 . Hence, there exists 𝑥 ∈ Γ 𝜈 and a neighborhood Ω ∈ R 𝑁 0 of 𝑥 such that spaces Γ+ 𝜈 = {𝑥 ∈ R 𝑁 0 | ⟨𝑤 •→𝜈 , 𝑥⟩ + 𝑏 𝜈 > 0} and similarly for Γ - 𝜈 . Let 𝑥 + ∈ Ω ∪ Γ + 𝜈 and 𝑥 -∈ Ω ∪ Γ - 𝜈 .We have 𝑎 𝜈 (𝑥 + ) = 1 and 𝑎 𝜈 (𝑥 -) = 0. Moreover, for all 𝜈 ′ ̸ = 𝜈, we have 𝑎 𝜈 (𝑥 + ) = 𝑎 𝜈 (𝑥 -). We evaluate Equation (A.157) on 𝑥 + and 𝑥 - and subtract both equations to get ⟨ 𝑥 --𝑥 + , 𝑧 ⟩ = ⟨ 𝑑 𝜈 , 𝑥 + ⟩ + 𝑒 𝜈 (A.159)

Figure B- 2 :

 2 Figure B-2: Energy of the network through successive ENorm iterations (without training). One color denotes one iteration. The darker the color, the higher the iteration number.

  [Theorem 4.1] 1 . Theorem B.2.1. Let 𝐷 ⊂ R 𝑛 be an open set and 𝑓 : 𝐷 → R a real function of 𝐵 block variables 𝑥 𝑏 ∈ R 𝑛 𝑏 with ∑ 𝐵 𝑏=1 𝑛 𝑏 = 𝑛. Let 𝑥 (0) be the starting point of the coordinate descent algorithm and 𝑋 the level set 𝑋 = { 𝑥 | 𝑓 (𝑥) ≤ 𝑓

  𝜙 is differentiable on the open set 𝐷 = (0, +∞) 𝑛 . Assumption (2). 𝜙 → +∞ when ‖𝑑‖ → +∞. Let 𝑑 such that 𝜙(𝑑) < 𝑀 𝑝 , 𝑀 > 1. Let us show by induction that for all ℓ ∈ 1, 𝐿 -1 , ‖𝑑 ℓ ‖ ∞ < (𝐶𝑀 ) ℓ , where 𝐶 = max(𝐶 0 , 1) and 𝐶 0 = max 𝑊(ℓ) [𝑖,𝑗]̸ =0

𝑑

  𝑘 [𝑗] 𝑑 𝑘-1 [𝑖] ) 𝑝 ⃒ ⃒ ⃒𝑊 (ℓ)[𝑖, 𝑗]

  minimum as shown in Section 4.3.3, see Equation (

𝐸 0 =

 0 {(ℓ, 𝑖, 𝑗) | 𝑊

  Algorithm 2, we perform alternative SGD and ENorm steps during training. We now derive an implicit formulation of this algorithm that we call Implicit Equi-normalization. Let us fix 𝑝 = 2. We denote by 𝒞(𝑅 𝜃 (𝑥), 𝑦) the cross-entropy loss for the training sample (𝑥, 𝑦) and by 𝐿 2 (𝜃, 𝑑) the weight decay regularizer (B.13). The loss function of the network writes ℒ(𝜃, 𝑑) = 𝒞(𝑅 𝜃 (𝑥), 𝑦) + 𝜆𝐿 2 (𝜃, 𝑑) (B.14)

  Figure B-3: ENorm cycles on a randomly initialized ResNet-18 with no training.

Figure B- 4 :

 4 Figure B-4: Training a fully-connected network on CIFAR-10, with (ENorm-1) or without (Baseline) Equi-normalization.

  Figure B-6: Uniform vs adaptive scaling on MNIST, with BN.

Figure C- 1 :

 1 Figure C-1: Effect of Quantization Parameters. We report the influence of the proportion of blocks to which we apply the noise. We focus on Transformer for Wikitext-103 language modeling. We explore two settings: iPQ and int8. For iPQ, we use 𝜙 proxy .

  Figure C-2: Quantizing with a larger number of centroids. Results are shown on Wikitext-103 valid.

  Figure C-4: Effect of Quantization on Model Structures. Results are shown on the validation set of Wikitext-103. (a) Quantizing Attention, FFN, and Embeddings in different order. (b) More Extreme compression of different structures.

Figure D- 1 :

 1 Figure D-1: Qualitative results using Motion based variants on mobile architectures, using a 32 × 32 × 256 latent space. The model generates the face given the fixed source frame and the landmarks of the target frame. All models run in real-time on an iPhone 8.

  . Source code: https://github.com/pytorch/fairseq/tree/master/examples/quant_ noise • Maxime Oquab * , Pierre Stock * , Oran Gafni, Daniel Haziza, Tao Xu, Peizhao Zhang, Onur Celebi, Yana Hasson, Patrick Labatut, Bobo Bose-Kolanu, Thibault Peyronel, Camille Couprie. Low Bandwidth Video-Chat Compression using Deep Generative Models. Under review, 2021

  First, BN normalizes 𝑥 per-channel into ̂ 𝑥. More formally, denoting 𝑐 ∈ 1, 𝐶 a given channel, we have

̂ 𝑥[:, 𝑐, :, :] = 𝑥[:, 𝑐, :, :] -𝜇 𝑐

  Table2.1. Since the considered activation functions are not homogeneous 28 , rescalings are replaced by sign flips. All authors consider networks with one output node. Recall that a parameterization is admissible if for every hidden neuron 𝜈 ∈ 𝐻 there exists a path 𝑝 ∈ 𝒫 going through 𝜈 such that 𝑣 𝑝 (𝜃) > 0 as explained in the Notations. We denote by 𝑤 •→𝜈 the vector (𝑤 𝜇→𝜈 ) 𝜇∈𝑁 0 . Thus, for a given architecture 𝐺 valued with 𝜃 = (𝑤, 𝑏), denoting the output neuron 𝜂, the function implemented by the network writes:

Table 2

 2

	.2.

  3) There exists 𝜇 1 , 𝜇 2 ∈ 𝑁 0 such that for all 𝜈 ∈ 𝑁 1 , 𝑤 𝜇 1 →𝜈 > 0 and 𝑤 𝜇 2 →𝜈 > 0. Let 𝐺 be a fixed, one-hidden layer architecture. Let 𝜃 and 𝜃 ′ be non-degenerate parameterizations on 𝐺, without biases. Then, there exists a non-degenerate parameterization 𝜃 ′′ such that 𝑅 𝜃 ′ = 𝑅 𝜃 ′′ and

	Assumption B.1 is similar to assumptions S.2, A.2 and K.2. Assumption B.2 refers to
	the notion of balanced network (see Subsection 2.2.5 and Chapter 4). Assumption B.3
	seems more restrictive. The statement is the following (Berner et al., 2019, Theorem
	3.3) .

  Thus, |𝑅 𝜃 ′′ -𝑅 𝜃 | Finally, we list below two interesting papers that closely relate to our investigations in Section 3.4.4 where we focus on local identifiability. 𝑥 ∈ R 𝑁 0 and 𝑦 ∈ R 𝑁 𝐿 an input/output pair and 𝜃 and 𝜃 ′ two parameterizations such that the considered network is indeed a deep structured network. Denote 𝛿 = ‖𝑦 -𝑅 𝜃 (𝑥)‖ and 𝜂 = ‖𝑦 -𝑅 𝜃 ′ (𝑥)‖. Then local stability writes as 𝑑(𝜃, 𝜃 ′ ) ≤ 𝐶(𝛿 + 𝜂) where 𝐶 ≥ 0 (Informal Theorem 1.1 in the paper).

	• Malgouyres and Landsberg (2018) consider so called Deep Structured Neural
	Network of arbitrary depth, without biases. Such linear networks are simply
	described by a product of weight matrices 𝑀 𝑖 (ℎ 𝑖 ), where each matrix depends
	linearly on parameters ℎ 𝑖 ∈ R 𝑆 for a given 𝑆 ∈ N. As explained in the pa-
	per, such architectures present strong connections with feedforward non-linear
	networks using the ReLU (for a fixed input 𝑥). Next, the authors define the
	rescaling equivalence by considering only inter-layer rescalings (Equation 3.2
	in the paper), whereas we consider more generic per-neuron rescalings in this
	manuscript, see for instance Definition 2.2.3. The authors seek to establish
	sufficient and necessary conditions on deep structured linear neural networks
	guaranteeing local stability. More formally, let 𝑑 be a metric taking into ac-
	count the inter-layer rescaling,

Table 2 .

 2 typically rely on a combination of 3: Respective number of parameters and multiplications for various types of convolutions. The input activations have size 1 × 𝐶 in × 𝐻 × 𝑊 . For depth-wise convolutions, we assume that 𝐶 in = 𝐶 out . Thus, sequentially combining one pointwise convolution with 𝐶 in input channels and 𝐶 out output channels, one depth-wise convolution and one point-wise convolution with 𝐶 out input channels and 𝐶 in output channels only necessitates a fraction 𝐶 out /𝐶 in (2/9 + 1/𝐶 in ) of the parameters and operations required by a single standard 3 × 3 convolution with 𝐶 in input channels and 𝐶 in output channels.

	Operation	Kernel Size Parameters Multiplications
	Standard Convolution	3 × 3	9𝐶 in 𝐶 out	𝑂(9𝐻𝑊 𝐶 in 𝐶 out )
	Point-wise Convolution	1 × 1	𝐶 in 𝐶 out	𝑂(𝐻𝑊 𝐶 in 𝐶 out )
	Depth-wise Convolution	3 × 3	9𝐶 out	𝑂(9𝐻𝑊 𝐶 out )

  The breakpoint is not visible on the realization 𝑅 𝜃 . In other words, there is no visible gradient change in the realization 𝑅 𝜃 , although such changes are visible on some intermediate realizations 𝑦 𝜈 for hidden neurons 𝜈 ∈ 𝐻. Let us assume that 𝜃 has two invisible neurons 𝜈 1 and 𝜈 2 as illustrated in Figure3-4. Let 𝜀 > 0. Since the breakpoint is not visible, we can translate the functions 𝑦 𝜈 1 and 𝑦 𝜈 2 on the horizontal axis (offsets 𝑏 ′ 𝜈 1 = 𝑏 𝜈 1 + 𝜀 and 𝑏 ′ 𝜈 2 = 𝑏 𝜈 2 + 𝜀) and translate 𝑅 𝜃 ′ on the vertical axis (offset 𝑏 ′ 𝜂 = 𝑏 𝜂 -𝜀) to compensate. Then, 𝑅 𝜃 = 𝑅 𝜃 ′ but 𝜃 and 𝜃 ′ are not permutation-rescaling equivalent.

Remark 3.3.2. Let

  Definition 3.3.4 (Trajectory rescaling equivalence). 𝜃 and 𝜃 ′ are rescaling equivalent if there exists 𝛾 ∈ Comp 𝜃 such that 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 . 𝜃 be an admissible parameterization. Consider 𝜃 ′ ∈ Sign 𝜃 and 𝛾 the unique perturbation associated with 𝜃 such that 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 (Proposition 3.3.5). 1. Using Proposition 3.1.1, we have 𝜃 ′ ∼ 𝑆 𝜃 if, and only if, 𝛾 ∈ Comp 𝜃 . 2. If 𝛾 ∈ Comp 𝜃 , 𝜃 ′ ∼ 𝑆 𝜃 and using Proposition 3.1.2, we have 𝑅 𝜃⊙𝑒 𝛾 = 𝑅 𝜃 .

  Definition 3.4.2. Consider 𝜃 an admissible parameterization. 𝜃 is restricted locally non-identifiable if for all neighborhood Ω of 𝜃, there exists 𝜃 ′ ∈ Ω ∩ Sign 𝜃 such that 𝑅 𝜃 ′ = 𝑅 𝜃 and 𝜃 ′ ̸ ∼ 𝑆 𝜃. Let 𝜃 be restricted locally non-identifiable. Then, 𝜃 is locally nonidentifiable. Moreover, if 𝜃 has full support, i.e. supp(𝜃) = R 𝐸 and if Sign 𝜃 is open, then 𝜃 is locally non-identifiable if, and only if, it is restricted locally non-identifiable.

	Proposition 3.4.1.

  Table 4.6.3. ENorm + BN outperforms all other methods, including WN + BN, by a good margin. Note that here ENorm refers to ENorm in the uniform

	Method	Average train 𝐿 2 error	Method	Test top 1 accuracy
	Baseline	0.542	Baseline	88.94
	BN	0.171	BN	90.32
	GN	0.171	GN	90.36
	WN + BN	0.162	WN + BN	90.50
	ENorm	0.179	ENorm	89.31
	ENorm + BN	0.102	ENorm + BN	91.35
	Table 4.2: Left: MNIST auto-encoder results (lower is better). Right: CIFAR-10
	fully convolutional results (higher is better).
	setup with the parameter 𝑐 = 1.2 whereas ENorm + BN refers to the uniform setup
	with 𝑐 = 1. A parameter study for different values and setups of the asymmetric
	scaling can be found in Appendix B.5.2.

Table 4

 4

	Batch size	32	64	128	256
	Baseline	66.20 68.60 69.20 69.58
	BN	68.01 69.38 70.83 71.37
	GN	68.94 68.90 70.69 70.64
	ENorm-1 (ours) 69.70 70.10 71.03 71.14

.3. The performance of BN decreases with small batches, which concurs with prior observations

[START_REF] Wu | Group normalization[END_REF]

. Our method outperforms GN and BN for batch sizes ranging from 32 to 128. GN presents stable results across batch sizes. Note that values of 𝑐 different from 1 did not yield better results. The training curves for a batch size of 64 are presented in Figure

4-5

. While BN and GN are faster to converge than ENorm, our method achieves better results after convergence in this case. Note also that ENorm overfits the training set less than BN and GN, but more than the Baseline case.

Table 4 .

 4 3: ResNet-18 results on the ImageNet dataset (test accuracy).

  architectures and compare our results to

	Model (original top-1) Compression Size ratio Model size	Top-1 (%)
	ResNet-18 (69.76%)	Small blocks Large blocks	29x 43x	1.54 MB 65.81 ±0.04 1.03 MB 61.10 ±0.03
	ResNet-50 (76.15%)	Small blocks Large blocks	19x 31x	5.09 MB 73.79 ±0.05 3.19 MB 68.21 ±0.04
	Table 5.1: Results for vanilla ResNet-18 and ResNet-50 for 𝑘 = 256 centroids.

Table 5

 5 Compression results for ResNet-18 and ResNet-50 architectures. We explore two compression regimes as defined in Section 5.4.1: small block sizes (block sizes of 𝑑 = 4 and 9) and large block sizes (block sizes 𝑑 = 8 and 18). The results of our method for 𝑘 = 256 centroids are of practical interest as they correspond to a byte-compatible compression scheme.

	ResNet-18 on ImageNet	ResNet-50 on ImageNet
	Test top-1 (%)	Test top-1 (%)
	Ours, small blocks	Ours, small blocks
	Ours, large blocks	Ours, large blocks
	Reference methods	Reference methods
	Original model		Original model
	Compression factor (original network size: 44.6MB)	Compression factor (original network size: 97.5MB)
	Figure 5-3: Size budget Best previous published method Ours
	∼1 MB	70.90% (HAQ , MobileNet v2) 64.01% (vanilla ResNet-18)
	∼5 MB	71.74% (HAQ, MobileNet v1)	76.12% (semi-sup. ResNet-50)
	∼10 MB	75.30% (HAQ, ResNet-50)	77.85% (semi-sup. ResNet-50)

.2: Best test top-1 on ImageNet for a given size (no architecture constraint). compression regime (block sizes of 4 and 9), with 𝑘 = 256 centroids (practical for implementation), our compressed semi-supervised ResNet-50 reaches 76.12% top-1 accuracy. In other words, the model compressed to 5.20 MB has the performance of a vanilla, non-compressed ResNet50 (vs. 97.5 MB for the non-compressed ResNet-50).

Table 5

 5

	.4.3. We argue that this

Table 5

 5

.4: Compression results for Mask R-CNN (backbone ResNet-50 FPN) for 𝑘 = 256 centroids (compression factor 26×).

  . A codebook is a set of 𝐾 vectors, i.e., 𝒞 = {𝑐[1], . . . , 𝑐[𝐾]}. Quantization methods compress the matrix 𝑊 by assigning to each block 𝑏 𝑘𝑙 an index that points to a codeword 𝑐 in a codebook 𝒞, and storing the codebook 𝒞 and the resulting indices (as the entries 𝐼 𝑘𝑙 of an index matrix 𝐼) instead of the real weights. During the inference, they reconstruct an approximation ̂ 𝑊 of the original matrix 𝑊 such that ̂ 𝑏 𝑘𝑙 = 𝑐[𝐼 𝑘𝑙 ].

Comparison of different quantization schemes with and without

  Table 6.1:

	Quantization Scheme	Language Modeling	Image Classification
		16-layer Transformer		EfficientNet-B3
			Wikitext-103			ImageNet-1k
		Size Compression PPL	Size Compression Top-1
	Uncompressed model	942	× 1	18.3	46.7	× 1	81.5
	int4 quantization	118	× 8	39.4	5.8	× 8	45.3
	-with QAT	118	× 8	34.1	5.8	× 8	59.4
	-with Quant-Noise	118	× 8	21.8	5.8	× 8	67.8
	int8 quantization	236	× 4	19.6	11.7	× 4	80.7
	-with QAT	236	× 4	21.0	11.7	× 4	80.8
	-with Quant-Noise	236	× 4	18.7	11.7	× 4	80.9
	iPQ	38	× 25	25.2	3.3	× 14	79.0
	-with QAT	38	× 25	41.2	3.3	× 14	55.7
	-with Quant-Noise	38	× 25	20.7	3.3	× 14	80.0
	iPQ & int8 + QNoise	38	× 25	21.1	3.1	× 15	79.8

Decomposing the impact of the different compression schemes. (a) we

  train Transformers with Adaptive Input and LayerDrop on Wikitext-103 (b) we pretrain RoBERTA base models with LayerDrop and then finetune on MNLI (c) we train an EfficientNet-B3 on ImageNet. We report the compression ratio w.r.t. to the original model ("Comp.") and the resulting size in MB.

		Language modeling	RoBERTa	Image Classification
		Comp. Size PPL	Comp. Size Acc.	Comp. Size	Acc.
	Unquantized
	Original	× 1.0 942	18.3	× 1.0 480 84.8	× 1.0 46.7	81.5
	+ Sharing	× 1.8 510	18.7	× 1.9 250 84.0	× 1.4 34.2	80.1
	+ Pruning	× 3.7 255	22.5	× 3.8 125 81.3	× 1.6 29.5	78.5
	Quantized
	iPQ	× 24.8 38	25.2	× 12.6 38 82.5	× 14.1 3.3	79.0
	+ Quant-Noise	× 24.8 38	20.7	× 12.6 38 83.6	× 14.1 3.3	80.0
	+ Sharing	× 49.5 19	22.0	× 34.3 14 82.5	× 18.0 2.6	78.9
	+ Pruning	× 94.2 10	24.7	× 58.5 8 78.8	× 20.0 2.3	77.8
	Table 6.2:

  Table6.3: Quant-Noise: Finetuning vs training. We report performance after iPQ quantization. We train with the 𝜑 proxy noise and finetune with Quant-Noise, and use it during the transfer to MNLI for each RoBERTa model.

	Language Modeling	PPL	RoBERTa	Acc.
	Train without Quant-Noise	25.2	Train without Quant-Noise	82.5
	+ Finetune with Quant-Noise	20.9	+ Finetune with Quant-Noise	83.4
	Train with Quant-Noise	20.7	Train with Quant-Noise	83.6

  Table7.1: Comparison of our approaches running on mobile in terms of compression for both model size and stream."10U" (resp. "20L") means that 10 unsupervised keypoints (resp. 20 facial landmarks) are used as inputs to the dense motion network. SPADE variants require 60 extra facial landmarks to draw the facial label maps. Notes: the "int8 size" is the full combined size of the models. The number of frames per second (FPS) is measured for the whole int8-quantized pipeline running on an iPhone 8, including landmark detection, grid-samples and face alignment. The #FLOPS count is for the dense motion, decoder, and unsupervised keypoint extractor networks. The bandwidth (BW) is measured at 25 FPS with Huffman encoding.

	Model variant	Inputs	FPS Params FLOPS int8 size	BW
	Motion Net	10U	18	2.9 M 1411 M 3.1 MB 1.4 kbits/s
	Motion Net	20L	19	2.3 M 1293 M 2.5 MB 2.2 kbits/s
	Motion Net	10U + 20L 14	3.0 M 1505 M 3.4 MB 3.6 kbits/s
	Motion SPADE 10U	16	2.9 M 1198 M 3.2 MB 8.0 kbits/s
	Motion SPADE 20L	19	2.3 M 1029 M 2.5 MB 8.8 kbits/s
	Motion SPADE 10U + 20L 13	3.0 M 1292 M 3.4 MB 10.2 kbits/s

Table 7 .

 7 3 provides a quality assessment of different models by human raters. Participants are asked to rate images produced by the different models by comparingLPIPS ↓ NME ↓ CSIM ↑Table 7.2: Evaluation results for Motion Net approaches without adversarial finetuning on the VoxCeleb2-28 video subset. Mob : Mobile models. Dense models (64 × 64 latent space) are trained on VoxCeleb. Mobile models (32 × 32) are trained on the DFDC aligned dataset. U: unsupervised keypoints; L: facial landmarks.

	Dense MN-10 U	0.221	0.59	0.83
	Dense MN-20 L	0.242	0.50	0.80
	Dense MN-68 L	0.240	0.49	0.81
	Mob MN-10 U	0.225	0.52	0.79
	Mob MN-20 L	0.244	0.48	0.78
	Mob MN-10 U + 20 L	0.218	0.46	0.80
	Mob M-SPADE-10 U	0.217	0.47	0.81
	Mob M-SPADE-20 L	0.242	0.44	0.79
	Mob M-SPADE-10 U + 20 L	0.215	0.46	0.81

Table 7 .

 7 . 3: Quality assessment of different dense models on DFDC-50 -Human study.

		Dense models
		human identity human expression
	model	score		score
	NTH	3.71±0.041	3.77±0.040
	Bilayer	3.70±0.046	3.62±0.046
	SegFace	3.11±0.047	3.00±0.051
	FOM adv	3.99±0.042	4.00±0.041
	Overall human ratings of Mobile models
		MN-10	MN-20 MN-10+20
	no SPADE	3.44±0.034 3.40±0.034	3.46±0.034
	with SPADE 3.50±0.034 3.46±0.035	3.45±0.034

  =⇒ Layer. If 𝜃 ∼ Trajectory 𝜃 ′ , there exists a 𝜃-rescaling-compatible 𝛾 = (𝛼, S𝛽) ∈ Supp 𝜃 such that 𝜃 ′ = 𝜃 ⊙ 𝑒 𝛾 . Let us define𝛿 = exp(S𝛼) ∈ R 𝐻 . (A.10)Note that 𝛿 is well-defined since 𝛼 ∈ ker (P 𝜃 ). Let 𝜈 ∈ 𝐻 such that 𝑏 𝜈 ̸ = 0 and let 𝑞 be a partial path going from 𝜈 to the output neuron such that 𝑣 𝑞 (𝜃) ̸ = 0, possible since𝜃 is admissible. Let us show that (S𝛽) 𝜈 = (S𝛼) 𝜈 . Recalling that 𝛼 -𝛽 ∈ ker (Q 𝜃 ),𝑏 𝑞 0 𝑣 𝑞 (𝜃) 𝑏 𝑞 0 𝑣 𝑞 (𝜃) ̸ = 0, we deduce (S𝛽) 𝜈 = (S𝛼) 𝜈 . Next, let us define for every hidden layer 1 ≤ ℓ ≤ 𝐿 -1,𝐷 (ℓ) = Diag(𝛿 𝜈 ) 𝜈∈𝑁 ℓ . (A.12) Next, recall that 𝑤 ′ = 𝑤 • exp(𝛼). Let 𝑒 = 𝜇 → 𝜈 ∈ 𝐸. • If 𝜇 ∈ 𝑁 0 , let us show that 𝑤 ′ 𝑒 = 𝑤 𝑒 𝛿 𝜈 for every 𝑒 ∈ 𝐸 such that 𝑤 𝑒 ̸ = 0. First, We have 𝑤 ′ 𝑒 = 𝑤 𝑒 exp(𝛼 𝑒 ). Moreover, 𝑞 is any partial path going from 𝜈 to the output neuron such that 𝑣 𝑞 (𝜃) ̸ = 0. Since 𝑝 = (𝜇) + 𝑞 (where + denotes path concatenation) is also such that 𝑣 𝑝 (𝜃) ̸ = 0, and since 𝛼 ∈ ker P 𝜃 , we have 𝜈 = 𝛼 𝑒 and 𝑤 ′ 𝑒 = 𝑤 𝑒 exp(𝛼 𝑒 ) = 𝑤 𝑒 exp((S𝛼) 𝜈 ) = 𝑤 𝑒 𝛿 𝜈 .

	∑ 𝑒∈𝑞 Thus, since (S𝛼) 𝜈 = 𝛼 𝑒 = 𝑏 𝑞 0 𝑣 𝑞 (𝜃) ∑ 𝑒 ′ ∈𝑞 𝛼 𝑒 ′ ∑ 𝑒∈𝑞 ∑ 𝑒 ′ ∈𝑝 𝛼 𝑒 ′ = 𝛼 𝑒 + ∑ 𝑒 ′ ∈𝑞 𝛼 𝑒 ′ = 𝛼 𝑒 -(S𝛼) 𝜈 𝛽 𝑒 . where 0 = Thus, (S𝛼)	(A.11) (A.13) (A.14)

3 One Hidden Layer Case Proof of Proposition 3.2.1 Proposition.

  Thus, there exists a parameterization 𝜃 ′′′ such that 𝜃 ∼ 𝑆 𝜃 ′′′ and 𝜃 ′′′ ∼ 𝑃 𝜃 ′ , i.e. 𝜃 and 𝜃 ′ are rescaling-permutation equivalent. The reciprocal and the case with biases is similar and is not displayed here. 𝑃 𝜃 ′′ and 𝜃 ′′ ∼ 𝑆 𝜃 ′ . Using Propositions 164 A.1.2, we have 𝑅 𝜃 = 𝑅 𝜃 ′′ and 𝑅 𝜃 ′′ = 𝑅 𝜃 ′ . Thus, 𝑅 𝜃 = 𝑅 𝜃 ′′ . Let 𝐺 be a one-hidden layer architecture valued with 𝜃 and 𝜃 ′ . Assume that 𝜃 and 𝜃 ′ are irreducible. Then, 𝑅 𝜃 = 𝑅 ′ 𝜃 implies that 𝜃 ∼ 𝑃 𝑆 𝜃 ′ . Proof. Let 𝜃 and 𝜃 ′ be two irreducible parameterizations such that 𝑅 𝜃 = 𝑅 𝜃 ′ . Recall that 𝑥 ↦ → 𝑅 𝜃 (𝑥) and 𝑥 ↦ → 𝑅 𝜃 ′ (𝑥) are functions from R 𝑁 0 to R 𝑁 2 . Let us show that 𝜃 ∼ 𝑃 𝑆 𝜃 ′ . Since 𝑅 𝜃 = 𝑅 𝜃 ′ , for any output neuron 𝜂 and 𝑥 ∈ R 𝑁 0 ,

	∑
	𝜈∈𝑁 1
	Proof of Proposition 3.1.2
	Proposition. Two parameterizations that are permutation-rescaling equivalent are
	functionally equivalent.

Proof. Let 𝜃 and 𝜃 ′ be permutation-rescaling equivalent parameterizations. There exists a parameterization 𝜃 ′′ such that 𝜃 ∼ A.1.

  ∑𝜈∈𝒥 𝑎 𝜈 𝑠 𝜈 ̸ = 0. This comes from Equation (A.39) and from the fact that𝑎 𝜈 𝑠 𝜈 = 𝑤 𝜈→𝜂 ⟨𝑤 •→𝜈 , 𝑢⟩𝑠 2 𝜈 = 𝑤 𝜈→𝜂 ⟨𝑤 •→𝜈 , 𝑢⟩.We are now ready to apply the identification Lemma A.1.1 stated below.

Lemma A.1.1. Let ℐ, ℐ ′ two finite sets of indices, empty or not 2 . For 𝑖 ∈ ℐ, let

𝑎 𝑖 ∈ R * , 𝑠 𝑖 ∈ {-1, +1}, 𝑡 𝑖 ∈ R such that the 𝑡 𝑖 are distinct and 𝑐 ∈ R. Define 𝜙 as 𝜙 : 𝑡 ↦ → ∑ 𝑖∈ℐ 𝑎 𝑖 𝜎(𝑠 𝑖 (𝑡 -𝑡 𝑖 )) + 𝑐. (A.49)

Similarly, we define 𝑎 ′ 𝑖 , 𝑠 ′ 𝑖 , 𝑡 ′ 𝑖 for 𝑖 ∈ ℐ ′ , 𝑐 ′ and the function 𝜓. We further assume that, for all 𝒥 ⊂ ℐ,

  We still fix an output neuron 𝜂 ∈ 𝑁 2 . Let us show that 𝜋 is independent from the considered vectors 𝑢 ∈ 𝒰 and 𝑟 ∈ ℛ 𝑢 . To this end, we explicitly state the a priori dependency of 𝜋 on 𝑢 and 𝑟 by writing 𝜋 = 𝜋 𝑢,𝑟 . Let us first fix 𝑢 ∈ 𝒰 and show that the vectors 𝑤 ′•→𝜋𝑟,𝑢(𝜈) and 𝑤 •→𝜈 are collinear for all 𝑟 ∈ ℛ 𝑢 . Let 𝑟 ∈ ℛ 𝑢 . • First, since ℛ 𝑢 is open, there exists a neighborhood 𝑉 of 𝑟 such that 𝑉 ⊂ ℛ 𝑢 . • Second, 𝜋 𝑢,𝑟 is entirely determined by the relation 𝑡 ′ 𝜋𝑢,𝑟(𝜈) = 𝑡 𝜈 for all 𝜈 ∈ 𝑁 1 , hence by the relative ordering of the 𝑡 𝜈 and 𝑡 ′ 𝜈 . Since the 𝑡 𝜈 (resp. 𝑡 ′ 𝜈 ) are distinct and since 𝑡 𝜈 and 𝑡 ′ 𝜈 depend continuously on 𝑟, there exists a neighborhood 𝑉 ′ of 𝑟 such that, for all 𝑟 ′ ∈ 𝑉 ′ , the relative ordering of the 𝑡 𝜈 and 𝑡 ′ 𝜈 is preserved, hence 𝜋 𝑢,𝑟 ′ = 𝜋 𝑢,𝑟 . • We deduce from the first two points and from Equation (A.53) applied in 𝑟 and to any 𝑟 • Regarding the rescalings, Equation (A.67) implies that 𝑑 𝜈→𝜂 does not depend on 𝜂 in the sense that, if 𝜈 ∈ ℐ 𝜂 1 ∩ ℐ 𝜂 2 , 𝑑 𝜈→𝜂 1 = 𝑑 𝜈→𝜂 2 . Then, for all hidden neuron 𝜈 ∈ 𝑁 1 , there exists 𝜂 such that 𝜈 ∈ ℐ 𝜂 since there are no dead neurons. We then define 𝑑 𝜈 = 𝑑 𝜈→𝜂 . Let ℐ, ℐ ′ two finite sets of indices, empty or not 3 . For 𝑖 ∈ ℐ, let 𝑎 𝑖 ∈ R * , 𝑠 𝑖 ∈ {-1, +1}, 𝑡 𝑖 ∈ R such that the 𝑡 𝑖 are distinct and 𝑐 ∈ R. Define 𝜙 as 𝜙 : 𝑡 ↦ → The function 𝜙 is piecewise affine and has exactly|ℐ| distinct breakpoints because all the 𝑡 𝑖 are distinct and the 𝑠 𝑖 are non-zero. We denote this set of breakpoints 𝒯 = {𝑡 𝑖 } 𝑖∈ℐ . Similarly, the function 𝜓 has exactly |ℐ ′ | breakpoints. We denote this set of breakpoints 𝒯 ′ = {𝑡 ′ 𝑖 } 𝑖∈ℐ ′ . Note that 𝒯 may be empty since ℐ may be empty, and similarly for 𝒯 ′ .Then, because 𝜙 = 𝜓, |ℐ| = |ℐ ′ | and 𝒯 = 𝒯 ′ , otherwise there would exist one point where one function would be differentiable and the other not. Thus, up to a re-numbering of ℐ ′ , we assume ℐ = ℐ ′ .The re-ordering of the numbers 𝑡 𝑖 and 𝑡 ′ 𝑖 and the fact that 𝒯 = 𝒯 ′ implies that ∀𝑖 ∈ ℐ, 𝑡 𝑖 = 𝑡 ′ 𝑖 . Let us show that ∀𝑖 ∈ ℐ, 𝑎 𝑖 = 𝑎 ′ 𝑖 . We compute the derivative of 𝜙 in a neighbourhood Ω of 𝑡 𝑖 such that Ω ∩ 𝒯 = {𝑡 𝑖 }, which is possible because the 𝑡 𝑖 are distinct. For 𝑡 ∈ Ω∖{𝑡 𝑖 },𝜙 ′ (𝑡) = 𝑎 𝑖 𝑠 𝑖 1(𝑠 𝑖 (𝑡 -𝑡 𝑖 )) + 𝑓 (𝑡) (A.70)where the function 𝑓 is constant on Ω and where 1(𝑥) = 1 if 𝑥 > 0 and 0 otherwise. Similarly, for 𝑡 ∈ Ω∖{𝑡 𝑖 }, 𝜓 ′ (𝑡) = 𝑎 ′ 𝑖 𝑠 ′ 𝑖 1(𝑠 ′ 𝑖 (𝑡 -𝑡 ′ 𝑖 )) + 𝑔(𝑡). Then, we use the equality 𝜙

	Finally, we have shown that 𝜃 and 𝜃 ′ are permutation-rescaling equivalent accord-
	ing to Definitions 2.2.3 and 2.2.7.
	We finally prove Lemma A.1.1.
	Lemma. ∑
	•→𝜈 , 𝑢⟩ Then, using Equation (A.51) with Equation (A.52) and the fact that both 𝑠 𝜈 and (A.51) (A.52) (A.53) 𝑠 𝜋(𝜈) are non-zero, we deduce that for all 𝜈 ∈ ℐ, tion (A.55) is valid for any small enough direction 𝜏 , we deduce that 𝑤 ′ •→𝜋𝑢,𝑟(𝜈) = 𝑑 𝜈→𝜂,𝑢,𝑟 𝑤 •→𝜈 . (A.56) which proves that 𝑤 ′ •→𝜋𝑢,𝑟(𝜈) and 𝑤 •→𝜈 are collinear. Note that thanks to Equa-tion (A.54), we also have 𝑑 𝜈→𝜂,𝑢,𝑟 = 𝑤 𝜈→𝜂 𝑤 ′ 𝜋𝑢,𝑟(𝜈)→𝜂 . (A.57) Next, using Equations (A.53) and (A.56), we deduce that 𝑏 ′ 𝜋𝑢,𝑟(𝜈) = 𝑑 𝜈→𝜂,𝑢,𝑟 𝑏 𝜈 . (A.58) 𝑤 = 0. Since Equa-(A.59) Then, using Equations (A.56) and (A.58), for any 𝑟 ) . (A.60)

′ 𝜋(𝜈)→𝜂 ⟨𝑤 ′ •→𝜋(𝜈) , 𝑢⟩ = 𝑤 𝜈→𝜂 ⟨𝑤 •→𝜈 , 𝑢⟩. (A.54)

Step 3.

′ ̸ = 𝑟 ∈ 𝑉 ∩ 𝑉 ′ that ⟨𝑤 ′ •→𝜋𝑢,𝑟(𝜈) , 𝜏 ⟩ = 𝑑 𝜈→𝜂,𝑢,𝑟 ⟨𝑤 •→𝜈 , 𝜏 ⟩ (A.55)

where

𝜏 = 𝑟 ′ -𝑟 and 𝑑 𝜈→𝜂,𝑢,𝑟 = ⟨𝑤 ′ •→𝜋𝑢,𝑟(𝜈) , 𝑢⟩/⟨𝑤 •→𝜈 , 𝑢⟩ ̸ ′ ̸ = 𝑟 ∈ ℛ 𝑢 and for all 𝑥 ∈ R 𝑁 0 , ⟨𝑥, 𝑤 ′ •→𝜋 𝑢,𝑟 ′ (𝜈) ⟩ + 𝑏 ′ 𝜋 𝑢,𝑟 ′ (𝜈) = 𝑑 𝜈→𝜂,𝑢,𝑟 ′ (⟨𝑥, 𝑤 •→𝜈 ⟩ + 𝑏 𝜈 ) = 𝑑 𝜈→𝜂,𝑢,𝑟 ′ 𝑑 𝜈→𝜂,𝑢,𝑟 ( ⟨𝑥, 𝑤 ′ •→𝜋𝑢,𝑟(𝜈) ⟩ + 𝑏 ′ 𝜋𝑢,𝑟(𝜈) 𝑖∈ℐ 𝑎 𝑖 𝜎(𝑠 𝑖 (𝑡 -𝑡 𝑖 )) + 𝑐. (A.68)

Similarly, we define 𝑎 ′ 𝑖 , 𝑠 ′ 𝑖 , 𝑡 ′ 𝑖 for 𝑖 ∈ ℐ ′ , 𝑐 ′ and the function 𝜓. We further assume that, for all 𝒥 ⊂ ℐ, ∑ 𝑖∈𝒥 𝑎 𝑖 𝑠 𝑖 ̸ = 0. (A.69) We assume ∀𝑡, 𝜙(𝑡) = 𝜓(𝑡). Then, 𝑐 = 𝑐 ′ and up to a re-numbering, ℐ = ℐ ′ and for all 𝑖 ∈ ℐ, 𝑡 𝑖 = 𝑡 ′ 𝑖 , 𝑠 𝑖 = 𝑠 ′ 𝑖 and 𝑎 𝑖 = 𝑎 ′ 𝑖 .

Proof. We first show that |ℐ| = |ℐ ′ |.

  81)where ̃ 𝒫 = {(𝑝 0 , . . . , 𝑝 𝐿-1 ) | 𝑝 ∈ 𝒫} and ̃ 𝒬 ℓ = {(𝑝 ℓ , . . . , 𝑝 𝐿-1 ) | 𝑝 ∈ 𝒫}. We then observe that

	𝜎	(	̃ 𝑅 𝜃 (𝑥) 𝜇	)	= 1(𝑦 𝜇 (𝜃, 𝑥) > 0) ̃ 𝑅 𝜃 (𝑥) 𝜇 .	(A.82)
	We use both this fact and Equation (A.81) to rewrite Equation (A.80) as

  85)Then, 𝒳 𝜃 is open. Moreover, let 𝜃 be a parameterization and 𝑥 ∈ 𝒳 𝜃 . Then, for each full or partial path 𝑝 ∈ 𝒫 ∪ 𝒬, the function 𝑥 ′ ↦ → 𝑎 𝑝 (𝜃, 𝑥 ′ ) (resp. 𝜃 ′ ↦ → 𝑎 𝑝 (𝜃 ′ , 𝑥)) is locally constant in the neighborhood of𝑥 (resp. in the neighborhood of 𝜃). . To show that 𝒳 𝜃 is open, consider 𝑥 ∈ 𝒳 𝜃 . By definition there exists 𝑈 ∈ 𝒰(𝑥, 𝜃) such that, for all 𝑝 ∈ 𝒫 ∪ 𝒬 and for all (𝑥 ′ , 𝜃 ′ ) ∈ 𝑈 , 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃 ′ , 𝑥 ′ ). Since 𝑈 is a neighborhood of (𝑥, 𝜃), there exists two open neighborhoods 𝑉 , 𝑊 of 𝑥 (resp. of 𝜃) such that 𝑉 × 𝑊 ⊂ 𝑈 . Considering 𝑥 ′ ∈ 𝑉 we now show that 𝑥 ′ ∈ 𝒳 𝜃 . This will imply that 𝒳 𝜃 is open as claimed. First, (𝑥 ′, 𝜃) ∈ 𝑈 , hence 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃, 𝑥 ′ ). Second, by definition of 𝑥 ∈ 𝒳 𝜃 , we have 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃 ′′ , 𝑥 ′′ ) for all 𝜃 ′′ , 𝑥 ′′ ∈ 𝑈 . Hence, noting that 𝑈 ∈ 𝒰(𝑥 ′ , 𝜃) since 𝑉 is a neighborhood of 𝑥 ′ , we deduce that for all 𝜃 ′′ , 𝑥 ′′ ∈ 𝑈 , we have 𝑎 𝑝 (𝜃, 𝑥 ′ ) = 𝑎 𝑝 (𝜃 ′′ , 𝑥 ′′ ).Next, let us show that the function 𝜃 ′ ↦ → 𝑎 𝑝 (𝜃 ′ , 𝑥) is locally constant. Let 𝑥 ∈ 𝒳 𝜃 . Then, there exists 𝑈 ∈ 𝒰(𝑥, 𝜃) such that, for all 𝑝 ∈ 𝒫 ∪ 𝒬 and for all (𝑥 ′ , 𝜃 ′ ) ∈ 𝑈, 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃 ′ , 𝑥 ′ ). As before, there are open neighborhoods 𝑉, 𝑊 such that 𝑉 ×𝑊 ⊂ 𝑈 where 𝑉 ⊂ 𝑅 𝑁 0 is a neighborhood of 𝑥 and 𝑊 ⊂ 𝑅 𝐸∪𝐻 is a neighborhood of 𝜃. Since 𝑥 ∈ 𝑉 , we have that for all 𝜃 ′ ∈ 𝑊 , 𝑎 𝑝 (𝜃, 𝑥) = 𝑎 𝑝 (𝜃 ′ , 𝑥). Thus, the function 𝜃 ↦ → 𝑎 𝑝 (𝜃, 𝑥) is locally constant. The proof that the function 𝑥 ↦ → 𝑎 𝑝 (𝜃, 𝑥) is locally constant is similar. The function 𝑥 ↦ → 𝑅 𝜃 (𝑥) is piecewise affine continuous and locally affine in the neighborhood of any 𝑥 ∈ 𝑋 𝜃 .Proof. The piecewise affine nature comes from Proposition 3.3.1, where we have𝑅 𝜃 (𝑥) 𝜈 = 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) + 𝑣 𝑞 (𝜃)𝑎 𝑞(𝜃, 𝑥). notice that 𝑥 ↦ → 𝐴 𝜇 (𝑥) and 𝑥 ↦ → 𝐵(𝑥) are piecewise constant using Proposition 3.3.2, hence the result. The continuity comes from Equation2.9 which shows that 𝑥 ↦ → 𝑅 𝜃 (𝑥) is the composition of continuous functions. For any 𝑥 ∈ 𝒳 𝜃 , the function 𝜃 ↦ → 𝑅 𝜃 (𝑥) is piecewise polynomial continuous and locally polynomial in the neighborhood of any 𝜃.𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) +We notice that 𝑥 ↦ → 𝐴 𝜇 (𝑥) and 𝑥 ↦ → 𝐵(𝑥) are piecewise constant using Proposition 3.3.2, hence the result. The continuity comes from Equation2.9 which shows that 𝜃 ↦ → 𝑅 𝜃 (𝑥) is the composition of continuous functions. Let 𝑒 ∈ 𝐸. Then, for all 𝑥 ∈ 𝒳 𝜃 , Let 𝑞 be a path going from some input neuron to 𝜈 and define 𝑝 = 𝑞 +𝑞 and 𝑝 ′ = 𝑞 +𝑞 ′ (recall that + denotes path concatenation). As 𝛼 ∈ ker P, we have 𝑝 = 0 = (P𝛼) 𝑝 ′ = Let 𝛼 ∈ ker P such that S𝛼 = 0. Considering any edge 𝑒 = 𝜇 → 𝜈 ∈ 𝐸 we show that 𝛼 𝑒 = 0, hence 𝛼 = 0. Let 𝑞 be a partial path going from 𝜈 to the output neuron and 𝑞 = {𝜇} + 𝑞. Then, 0

	with the following coefficients
				𝐴 𝜇 (𝑥) =	∑	𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥)
	𝑒∈𝑞 ∑	𝛼 𝑒 +	𝑒∈𝑞 ∑	𝛼 𝑒 =	∑	𝑝∈𝒫 𝑝 0 =𝜇 𝐿 𝑝 𝐿 =𝜈	𝑒∈𝑞 ∑	𝛼 𝑒 +	𝑒∈𝑞 ′ ∑	𝛼 𝑒 .
	𝑝∈𝒫 𝑝 𝐿 =𝜈 𝑒∈𝑝 𝑝 𝐿 =𝜈 𝑝∈𝒫 ∑ Proposition. ∑ ( 𝜕𝑅 𝜃 (𝑥) 𝜈 𝜕𝑤 𝑒 ) = Proposition A.2.1. 𝑤 𝑒 Proposition. ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 = ∑ 𝑝∈𝒫 𝑝 𝐿 =𝜈 with the following coefficients 𝑣 𝑝 (𝜃)𝐴 𝑝 (𝜃) + 𝐴 Proof of Proposition A.2.1 ProofProof of Proposition 3.3.3 𝐵(𝑥) = ∑ ℓ=1 ∑ ∑ 𝐿 ℓ=1 𝑞∈𝒬 ℓ→𝐿 𝑞 𝐿 =𝜈 𝑏 𝑞 0 We Proof of Proposition 3.3.4 ∑ 𝐿 ℓ=1 𝐿 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ→𝐿 𝑞∈𝒬 ℓ→𝐿 ∑ 𝑞 𝐿 =𝜈 ∑ 𝑞∈𝒬 ℓ→𝐿 𝑞 𝐿 =𝜈 𝑞 𝐿 =𝜈 𝑣 𝑞 (𝜃)𝐵 𝑞 (𝜃) Thus, ∑ 𝑒∈𝑞 𝛼 𝑒 =

𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥)

= ∑ 𝜇∈𝑁 0 𝑥 𝜇 𝐴 𝜇 (𝑥) + 𝐵(𝑥)

Proof. The piecewise polynomial nature comes from Proposition 3.3.1, where we have

𝑅 𝜃 (𝑥) 𝜈 = 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥) 𝑝 (𝜃) = 𝑥 𝑝 0 𝑎 𝑝 (𝜃, 𝑥) 𝐵(𝑥) = 𝑏 𝑞 0 𝑎 𝑞 (𝜃, 𝑥). 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥) + 𝐿 ∑ ℓ=1 ∑ 𝑞∈𝒬 ℓ 𝑞 𝐿-ℓ =𝜈 𝑒∈𝑞 𝑏 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥). (A.86) 𝑒∈𝑝 𝛼 𝑒 = (P𝛼)

∑ 𝑒∈𝑞 ′ 𝛼 𝑒 and S is well-defined.

S is injective.

  . For 𝜃 ′ ∈ Sign 𝜃 we have 𝜙 -1 𝜃 (𝜃 ′ ) = (𝛼, 𝛽) ∈ 𝑉 𝜃 where for any edge 𝑒 ∈ 𝐸 and hidden neuron 𝜈 ∈ 𝐻 𝛼 𝑒

Then, 1. The function 𝜙 𝜃 : 𝑉 𝜃 → Sign 𝜃 is a 𝒞 ∞ -diffeomorphism. 2

  91)Proof. Since sign(𝜙 𝜃 (𝛼, 𝛽)) = sign(𝜃) for any (𝛼, 𝛽), we have 𝜙 𝜃 (𝛼, 𝛽) ∈ Sign 𝜃 . Let us show that 𝜙 𝜃 is both surjective and injective with the given expression of 𝜙 -1 𝜃 . The fact that 𝜙 𝜃 and 𝜙 -1 𝜃 are 𝒞 ∞ is clear from their expressions.𝜙 𝜃 is injective. Consider (𝛼, 𝛽), (α, β) ∈ 𝑉 𝜃 and assume that 𝜙 𝜃 (𝛼, 𝛽) = 𝜙 𝜃 (α, β).

Consider 𝑒 ∈ 𝐸. If 𝑤 𝑒 = 0 then, since supp(𝛼) ⊂ supp(𝑤) and supp(α) ⊂ supp(𝑤), we get 𝛼 𝑒

  𝑛 , S𝛽 𝑛 ) ∈ Sincomp 𝜃 such that 𝑅 𝜃⊙𝑒 𝛾𝑛𝜏𝑛 = 𝑅 𝜃 .(A.136) By Proposition 3.3.8, Sincomp 𝜃 is compact, hence there exists a subsequence of 𝛾 𝑛 that converges to 𝛾 = (𝛼, S𝛽). Without loss of generality and to make notations lighter, we assume that the sequence 𝛾 𝑛 converges. Thus, lim 𝑛→∞ 𝛼 𝑛 = 𝛼, lim 𝑛→∞ 𝛽 𝑛 = 𝛽 and lim 𝑛→∞ 𝜏 𝑛 = 0.Denote 𝜌 𝛾,𝑥 (𝑡) = 𝑅 𝜃⊙𝑒 𝛾𝑡 (𝑥) and consider 𝑥 ∈ 𝒳 𝜃 . By Proposition 3.3.7, there is a neighborhood Ω of 𝑡 = 0 such that, for each 𝑡 ∈ Ω and each 𝑛 ∈ N,𝜌 𝛾𝑛,𝑥 (𝑡) = ∑ 𝑝∈𝒫 𝑥 𝑝 0 𝑣 𝑝 (𝜃)𝑒 𝑡(P𝛼𝑛)𝑝 𝑎 𝑝 (𝜃, 𝑥) 𝑣 𝑞 (𝜃)𝑒 𝑡(Q(𝛼𝑛-𝛽𝑛))𝑞 𝑎 𝑞 (𝜃, 𝑥) 𝛼 𝑛 , 𝛽 𝑛 ) ∈ 𝐵 𝜃 since 𝛾 𝑛 ∈ Sincomp 𝜃 . In particular, we obtain that for every 𝑛 ∈ N, 𝑡 ↦ → 𝜌 𝛾𝑛,𝑥 (𝑡) is 𝒞 ∞ in Ω.Consider 𝑛 large enough such that 𝜏 𝑛 ∈ Ω. Taylor-Lagrange's formula yields𝜉 𝑛 ∈ ]0, 𝜏 𝑛 [ ⊂ Ω such that 𝜌 𝛾𝑛,𝑥 (𝜏 𝑛 ) = 𝜌 𝛾𝑛,𝑥 (0) + 𝜏 𝑛 𝜌 ′ 𝛾𝑛,𝑥 (0) +Let us show that there exists 𝐶 ≥ 0 such that, for all 𝑛 large enough,|𝜌 ′′ 𝛾𝑛,𝑥 (𝜉 𝑛 )| ≤ 𝐶. (A.139)Differentiating 𝜌 1,𝑛 twice with respect to 𝑡 and evaluating at 𝜉 𝑛 ∈ Ω yields 𝑣 𝑝 (𝜃) ((P𝛼 𝑛 ) 𝑝 ) 2 𝑒 𝜉𝑛(P𝛼𝑛)𝑝 𝑎 𝑝 (𝜃, 𝑥) (A.140)We denote ‖P‖ = ‖P‖ 2,2 the spectral norm of P and similarly for Q. Recalling that||𝛾 𝑛 ‖ 2 = 1, we have ‖𝛼 𝑛 ‖ 2 ≤ 1. Then, for every path 𝑝 ∈ 𝒫, |(P𝛼 𝑛 ) 𝑝 | ≤ ‖P𝛼 𝑛 ‖ 2 ≤ ‖P‖‖𝛼 𝑛 ‖ 2 ≤ ‖P‖. (A.141) Moreover, since 0 < 𝜉 𝑛 < 𝜏 𝑛 ≤ 1, we use Equation (A.141) to deduce 𝑒 𝜉𝑛(P𝛼𝑛)𝑝 ≤ 𝑒 |𝜉𝑛(P𝛼𝑛)𝑝| ≤ 𝑒 |(P𝛼𝑛)𝑝| ≤ 𝑒 ‖P‖ . 𝑛 (𝜉 𝑛 ) is bounded from above by a term that is independent of 𝑛, namely 𝑝 0 𝑣 𝑝 (𝜃)𝑎 𝑝 (𝜃, 𝑥)| . (A.143) Similarly, we show that, for 𝑛 large enough, 𝑞 0 𝑣 𝑞 (𝜃)𝑎 𝑞 (𝜃, 𝑥)| . (A.144) Summing the right hand sides of (A.143)-(A.144) yields 𝐶 ≥ 0 (independent of 𝑛) such that (A.139) holds for 𝑛 large enough. Since 𝜌 𝛾𝑛,𝑥 (𝜏 𝑛 ) = 𝑅 𝐺 (𝜃 ⊙ 𝑒 𝛾𝑛𝜏𝑛 ))

	(𝛼 ⏟ where we used that (𝜏 2 ⏞ 𝜌 1,𝑛 (𝑡) + ∑ 𝑞∈𝒬 𝑏 𝑞 0 ⏟ 𝑛 2 𝜌 ′′ 1,𝑛 (𝜉 𝑛 ) = ∑ Thus, 𝜌 ⃒ ⃒ ⃒𝜌 ′′ 1,𝑛 (𝜉 𝑛 ) ⃒ ⃒ ⃒ ≤ ‖P‖ 2 𝑒 ‖P‖ ∑ 𝑥 𝑝 0 (A.142) ⏞ 𝜌 2,𝑛 (𝑡) (A.137) 𝜌 ′′ 𝛾𝑛,𝑥 (𝜉 𝑛 ). (A.138) ⃒ ⃒ ⃒𝜌 ′′ 2,𝑛 (𝜉 𝑛 ) ⃒ ⃒ ⃒ ≤ ‖Q‖ 2 𝑒 2‖Q‖ ∑
	𝑝∈𝒫
	 𝛼 =	1 𝜏	𝛼 ⊥	̃ 𝛽 =	1 𝜏	𝛽 ⊥	(A.130)

′′ 1,𝑝∈𝒫 |𝑥 𝑞∈𝒬 |𝑏

Table B .

 B 1: Uniform vs adaptive scaling, CIFAR-10 fully convolutional (uniform setup). We also test the adaptive setup (see TableB.1). Once again, the dynamics with or without BN are quite different. With or without BN, 𝑐 = 1.2 performs the best, which may be linked to the fact that the ReLUs are cutting energy during each forward pass. With BN, the results are less sensitive to the values of 𝑐.TableC.1: Compression of ResNet-50 with Quant-Noise. We compare toStock et al. (2019b) in both the small and large blocks regime. For fair comparison, we hold the compression rate constant. Quant-Noise provides improved performance in both settings.

	Setting	Model	Compression Top-1 Accuracy
	Small Blocks	Stock et al. (2019b) Quant-Noise	19x 19x	73.8 74.3
	Large Blocks	Stock et al. (2019b) Quant-Noise	32x 32x	68.2 68.8
	Method	Test top 1 accuracy
	ENorm uniform 𝑐 = 1		86.98
	ENorm uniform 𝑐 = 0.8		79.88
	ENorm uniform 𝑐 = 1.2		89.31
	ENorm adaptive		89.28
	ENorm + BN uniform 𝑐 = 1	91.85
	ENorm + BN uniform 𝑐 = 0.8	90.95
	ENorm + BN uniform 𝑐 = 1.2	90.89
	ENorm + BN adaptive		90.79

Table D .

 D 1: Ablation study for FOM on VoxCeleb2-28. MN: FOM without first order approximation nor adversarial fine-tuning. Table D.2: Evaluation results for Motion Net approaches without adversarial finetuning on the VoxCeleb2-28 video subset. Mob : Mobile models. Dense models (64 × 64 latent space) are trained on VoxCeleb. Mobile models (32 × 32) are trained on the DFDC aligned dataset. U: unsupervised keypoints; L: facial landmarks.Table D.1 obtained with the original code.

		FOM adv FOM w/o adv MN
	msVGG ↓	85.6	87.5		87.9
	LPIPS ↓	0.226	0.233		0.236
	NME ↓	0.51	0.53		0.54
	CSIM ↑	0.83	0.81		0.82
			LPIPS ↓ NME ↓ CSIM ↑
	Dense MN-10 U		0.221	0.59	0.83
	Dense MN-20 L		0.242	0.50	0.80
	Dense MN-68 L		0.240	0.49	0.81
	Mob MN-10 U		0.225	0.52	0.79
	Mob MN-20 L		0.244	0.48	0.78
	Mob MN-10 U + 20 L	0.218	0.46	0.80
	Mob M-SPADE-10 U	0.217	0.47	0.81
	Mob M-SPADE-20 L	0.242	0.44	0.79
	Mob M-SPADE-10 U + 20 L	0.215	0.46	0.81

  Table D.3: Comparison of Bilayer, SegFace (48x48), FOM adv, NTH in terms of quality / bandwidth (kbits/s with 25 fps) trade-offs on VoxCeleb2-28. * : Metrics were computed using ground truth backgrounds. We also include our best mobile model, Motion-SPADE-20L (MS20L) in the comparison.

		NTH Bilayer SegFace FOM MS20L
	msVGG* ↓ 56.3	68.6	84.4	58.7	57.9
	LPIPS* ↓ 0.165 0.200	0.304 0.153 0.167
	NME* ↓	0.38	0.55	0.55	0.50	0.44
	CSIM* ↑	0.83	0.85	0.76	0.87	0.84
	kbits/s ↓	9.7	9.7	18	4.0	8.8

Every 18 or

months according to a majority of the estimates. However, it is uncertain that this empirical law will hold in the future: with a characteristic scale of 5 nanometers of 2020 down to 3 and 2 nanometers in the next years, the transistors are now so small that they begin to experience quantum tunneling effects perturbing their functioning.2 https://www.ina.fr/video/I04275518, television interview in French by Michel Drucker.3 The Turing Award, usually considered as the highest distinction in Computer Science, was awarded to Yoshua Bengio, Geoffrey Hinton, and Yann LeCun in 2018 for their "conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing".

https://gdpr-info.eu/. It is a substantial update of the European Data Protection Directive passed in 1995 by the EU.

https://engineering.fb.com/2018/11/07/ml-applications/fbgemm/

See Figure10of[START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

For convolutions, 𝑁 ℓ-1 = 𝐾 2 𝐶 where 𝐾 is the kernel size and 𝐶 the number of input channels.

The input image is cropped with a random size and a random aspect ratio, and finally resized to the input size.

For instance, two convolutions interleaved with one ReLU.

Iterating over mini-batches of data, not single elements of the train set.

There is also a Nesterov version[START_REF] Nesterov | A method for solving the convex programming problem with convergence rate 𝑜(1/𝑘 2 )[END_REF] as well as the possibility to apply the learning rate directly to the gradient term ∇ 𝜃 ℒ instead of the velocity term 𝑣 𝑡+1 .

For instance, given a default value 𝜂 0 , use the learning rate 𝜂 0 /10 during the first 5 epochs (warm-up). Then, set the learning rate back to 𝜂 0 and decay it a factor 10 every 30 epochs (decay).

Generally, 𝜆 = 10 -4 or 10 -5 .

Such related work is in general anterior to[START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] that were the first to introduce the ReLU non-linearity and prove its effectiveness.

For instance, Definition 2.2.4 is not readily adaptable to sign flips.

By analogy with the brain, we might say that a neuron fires.

We denote by 𝑤 •→𝜈 the vector (𝑤 𝜂→𝜈 ) 𝜂∈𝑁0 .

[START_REF] Hanin | Deep relu networks have surprisingly few activation patterns[END_REF] show that Γ 𝜈 is a piecewise linear surface of codimension 1 for all but a measure-zero set of networks with respect to the Lebesgue measure.

More precisely, from (1) linear regions and (2) each affine function implemented by the network on each linear region. Note that knowing (1) + (2) is equivalent to knowing the realization 𝑅 𝜃 .

For instance, the weight 𝑤 𝜇2→𝜈2 can take an arbitrary value, more details in Section 3.2.

For instance, we can add any 𝜀 to 𝑤 𝜈1→𝜂 and subtract it from 𝑤 𝜈2→𝜂 , see Section 3.2.

In particular, without any constraint on the number of layers of 𝐺 and 𝐺 ′ or their sizes.

At that time, analogies to the human nervous system did not suggest that the ReLU non-linearity was a good candidate. It was considered later by[START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

𝑓 is homogeneous (of degree 1) if for every 𝑥 ∈ R and every 𝜆 > 0, 𝑓 (𝜆𝑥) = 𝜆𝑓 (𝑥).

The poles of 𝑧 ↦ → tanh(𝑧/2) are the points (2𝑚 + 1)𝑖𝜋 with 𝑚 ∈ Z.

Activation regions labelled 010, 110, 100 and 000 belong to the same linear region.

The complexity of the algorithm is not extensively studied by the authors for deep networks.

Although the underlying measure is not explicitly stated, we may assume that the authors consider a measure-zero set with respect to the Lebesgue measure. This measure-zero set is not explicitly described by the authors.

𝑊 1,∞ = 0 does not necessarily imply that 𝑅 𝜃 ′′ = 𝑅 𝜃 . We refer the reader toPetersen et al. (2018) for a counter-example of this notion of stability.

Thus not readily amenable to back-propagation.

We consider the weight 𝑤 𝑒 with 𝑒 ∈ 𝐸. Then, to prune 𝑤 𝑒 , we simply remove 𝑒 from the graph 𝐺. Equivalently, we set 𝑤 𝑒 = 0 and we do not update 𝑤 𝑒 during the backward pass. It amounts to detach the edge 𝑒 from the computational graph during the forward and backward passes.

The importance ranking of filters may differ when considering the 𝐿 1 or the 𝐿 2 norm for instance.

This claim implicitly assumes that the network is balanced, i.e. that the weights have the same order of magnitude. For the effect of rescalings on weight magnitude, see Chapter 4 and Figure4-1.

The Hessian has dimension: number of network parameters × number of network parameters, and is therefore a huge matrix, but it can be approximated to by a diagonal matrix, in which case it is no bigger than the gradient[START_REF] Ollivier | Riemannian metrics for neural networks I: feedforward networks[END_REF].

Related work around the so-called "Lottery Tickets" will not be discussed here.

On finite sets, provided that the dimension of 𝑦 is at least logarithmic in the size of the set.

A traditional building block generally includes a 3×3 convolution (i.e., a convolution with kernel size 3 × 3) interleaved with two 1 × 1 convolutions(He et al., 2015a).

Group convolutions with the number of groups equal to the number of input features, generally with a kernel size of 3 × 3.

Standard 1 × 1 convolutions.

The suffix 50 stands for the number of layers in the network.

The depth-wise convolution in Table2.3 is a particular case of group convolution.

Or averaged.

See in particularTable 3 of Wu et al. (2019a).

Also called float 32 or single-precision as opposed to fp64 double-precision.

The IEEE 754 fp32 format has 1 sign bit, 8 exponent bits and 23 mantissa bits to represent numbers in absolute value between 1 × 10 -38 and 3 × 10 38 with 6 to 9 significant decimal digits.

The IEEE 754 fp16 format has 1 sign bit, 5 exponent bits and 10 mantissa bits to represent numbers in absolute value between 6 × 10 -8 and 65,504 with 3 to 4 significant decimal digits.

This occupies additional memory in the GPU. However, the number of training activations is generally much higher than the number of weights in standard training regimes with large batches. Hence, this additional cost is minor.

https://github.com/NVIDIA/apex.

https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html.

For a square matrix of size |𝑁 ℓ | × |𝑁 ℓ |, it requires 2|𝑁 𝐿 | calibration parameters instead of 2.

https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py.

For instance, model the noise as a normal distribution 𝒩 (0, 𝜎) and compute the gradient wrt 𝜎.

Measure for instance by observing the loss degradation with quantizing only this particular layer.

For simplicity, we assume that 𝐶 in is a multiple of 𝑚, i.e. that all the subvectors have the same dimension 𝑑 = 𝐶 in /𝑚.

FLOPs denotes a number of floating point operations, whereas FLOPS denotes a number of floating point operations per second. Converting FLOPs to FLOPS requires a framerate, i.e. how many forward passes are required within one second.

Obtained using the framework: https://github.com/rwightman/pytorch-image-models.

We focus first on the one hidden layer case, for which we derive sufficient conditions such that functional equivalence implies permutation-rescaling equivalence and investigate their necessity. The considered general architecture is depicted in Figure3-1.

A proof does not seem to be present in the original paper.

Using quantities defined in the remainder of this Subsection.

Otherwise, we can multiply any zero weight or bias with any number without modifying it.

By convention, as explained in Definition

2.2.2, 𝐷(0) = 𝐼 𝑁0 and 𝐷(𝐿) = 𝐼 𝑁 𝐿 . 2 This corresponds respectively to the incoming and outgoing weights of a given neuron 𝜈.3 Each hidden neuron is connected to at least one input and one output neuron.

https://pytorch.org/docs/stable/torchvision/models.

The affine transform coefficients that converts an 8-bit quantized tensor (integer-valued in [0,

255]) to its floating-point counterpart.

We prove that the definitions of the rescalings and the permutation obtained in Step 2 are independent of the considered output neuron 𝜂, allowing to show that 𝜃 and 𝜃 ′ are indeed permutation and rescaling equivalent.

Recall that ⟨𝑤 •→𝜈1 , 𝑢⟩ ̸ = 0 and ⟨𝑤 •→𝜈2 , 𝑢⟩ ̸ = 0 using Equation (A.35).

We use the convention: ∑ ∅ = 0.

We use the convention: ∑ ∅ = 0.

Recall that we restrict ourselves to a scalar output with no output biases, see Subsection 3.3.2.

When valued with a particular parameterization 𝜃, this network is able to represent a triangular function R → R, hence the name.

In the BFS order.

Note that what Tseng denotes as stationary point in his paper is actually defined as a point where the directional derivative is positive along every possible direction, i.e. a local minimum.

Remerciements

Update momentum buffers with the same 𝐷 (ℓ) Appendix B.4, we also explore a method to jointly learn the rescaling coefficients and the weights with SGD, and report corresponding results.

Computational advantage over BN and GN. Table 4.1 provides the number of elements (weights or activations) accessed when normalizing using various techniques.

Assuming that the complexity (number of operations) of normalizing is proportional to the number of elements and assuming all techniques are equally parallelizable, we deduce that our method (ENorm) is theoretically 50 times faster than BN and 3 times faster than GN for a ResNet-18. In terms of memory, ENorm requires no extra-learnt parameters, but the number of parameters learnt by BN and GN is negligible (4800 for a ResNet-18 and 26,650 for a ResNet-50). We implemented ENorm using a tensor library. Taking full advantage of the theoretical reduction in compute would require to design an optimized CUDA kernel.

Model

Experiments

We analyze our approach by carrying out experiments on the MNIST and CIFAR-10 datasets and on the more challenging ImageNet dataset. ENorm will refer to Equi-normalization with 𝑝 = 2. [START_REF] Miguel | Model compression as constrained optimization, with application to neural nets[END_REF]. As we will see in the remainder of this Chapter, employing this discretization off-the-shelf does not optimize the right objective function, and leads to a catastrophic drift of performance for deep networks.

Pruning. Network pruning amounts to removing connections according to an importance criteria (typically the magnitude of the weight associated with this connection) until the desired model size/accuracy tradeoff is reached [START_REF] Lecun | Optimal brain damage[END_REF].

A natural extension of this work is to prune structural components of the network, for instance by enforcing channel-level (Liu et al., 2017b) or filter-level [START_REF] Luo | Thinet: A filter level pruning method for deep neural network compression[END_REF] sparsity. However, these methods alternate between pruning and re-training steps and thus typically require a long training time.

Dedicated Architectures. Architectures such as SqueezeNet [START_REF] Iandola | Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size[END_REF], NASNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF], ShuffleNet (Zhang et al., 2017b;[START_REF] Ma | Shufflenet V2: practical guidelines for efficient CNN architecture design[END_REF], Mo-bileNets (Sandler et al., 2018b) and EfficientNets [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF]) are designed to be memory efficient. As they typically rely on a combination of depth-wise and point-wise convolutional filters, sometimes along with channel shuffling, they are less prone than ResNets to structured quantization techniques such as PQ. These architectures are either designed by hand or using the framework of architecture search [START_REF] Howard | Searching for mobilenetv3[END_REF]. For instance, the respective model size and test top-1 accuracy of ImageNet of a MobileNet are 13.4 MB for 71.9%, to be compared with a vanilla ResNet-50 with size 97.5 MB for a top-1 of 76.2%. Moreover, larger models such as ResNets can benefit from large-scale weakly-or semi-supervised learning to reach better performance [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF][START_REF] Yalniz | Billionscale semi-supervised learning for image classification[END_REF].

Summary. Combining some of the mentioned approaches yields high compression factors as demonstrated by Han et al. (2016b) with Deep Compression (DC) or more recently by [START_REF] Tung | Deep neural network compression by in-parallel pruning-quantization[END_REF]. Moreover and from a practical point of view, the process of compressing networks depends on the type of hardware on which the networks will run. Recent work directly quantizes to optimize energy-efficiency and latency time on a specific hardware (Wang et al., 2018a). Finally, the memory overhead of storing the full activations is negligible compared to the storage of the weights for two reasons. First, in realistic real-time inference setups, the batch size is almost always equal to one. Second, a forward pass only requires to store the activations of the current layer -which are often smaller than the size of the input-and not the whole activations of the network.

Appendix A

Proofs for Functional Equivalence Classes

A.1 Permutation-Rescaling Equivalence Proof. We prove successive implications.

By assumption on the matrices 𝐷 (ℓ) , we have that 𝛾 𝜈 > 0 for every 𝜈 ∈ 𝐻. Let 𝜈 be a neuron in layer ℓ = 1. Since 𝑊 ′ (1) = 𝑊 (1) 𝐷 (1) , we have for any 𝜇 ∈ prev(𝜈)

Next, let 𝜈 be a neuron in layer 2 ≤ ℓ ≤ 𝐿-1 and 𝜇 ∈ prev(𝜈). Since

) -1 𝑊 (ℓ) 𝐷 (ℓ) , the following holds

Finally, let 𝜇 be a neuron in the last hidden layer 𝐿 -1 and 𝜈 ∈ next(𝜇). Since Proof. We show by induction that .16) for all ℓ ∈ 0, 𝐿 . We have 𝑦 (0) (𝜃 ′ , 𝑥) = 𝑥 = 𝑥𝐼 𝑁 0 = 𝑦 (0) (𝜃, 𝑥)𝐷 (0) and, if we assume Equation (A.16) valid for some ℓ ∈ 0, 𝐿 -2 , then using Definition 2.2.2,

) (ℓ+1) = 𝑦 (ℓ+1) (𝜃, 𝑥)𝐷 (ℓ+1) .

𝐷

(A.17)

For the last layer, as 𝐿) similarly yields Proof. We show by induction that

Proof of Proposition

for all ℓ ∈ 0, 𝐿 . We have (0) and, if we assume Equation (A.21) valid for some ℓ ∈ 0, 𝐿 -2 , then by Definition 2.2.6,

) (ℓ+1) = 𝑦 (ℓ+1) (𝜃, 𝑥)𝜋 (ℓ+1)

𝜋

The first relation comes from Equation (A.57), the second from Equation (A.56) and the third from Equation (A.58).

Step 4. We now prove that 𝜃 and 𝜃 ′ are permutation-rescaling equivalent. We explicitly denote by ℐ 𝜂 and ℐ ′ 𝜂 the sets constructed in Step 2 and similarly for the re-numbering 𝜋 𝜂 to show the dependency on the output neuron 𝜂. Let us show that the definitions of the rescaling coefficients 𝑑 𝜈→𝜂 and the permutations 𝜋 𝜂 in Step 2 are independent of the choice of 𝜂. Consider 𝜂 1 ̸ = 𝜂 2 two distinct output neurons.

• Regarding the permutation, we first show that if

(A.64) Equations (A.62) and (A.63) show that, for all 𝑥 ∈ R 𝑁 0 , where 𝑞 is the partial path going from 𝜈 to the output neuron 𝜂. Then, 1. S is well-defined (i.e., independent of the choice of 𝑞);

2. S is an isomorphism and its inverse S -1 : R 𝐻 → ker P is such that for any 𝜆 ∈ R 𝐻 we have S -1 𝜆 = 𝛽 where for any edge

Proof. Note that S is linear. The proof follows three steps.

S is well-defined. Let 𝛼 ∈ ker P. We wish to show that if 𝑞 and 𝑞 ′ are two partial paths going from 𝜈 to the output neuron 𝜂, then

Sincomp 𝜃 ⊂ Comp 𝜃 . Assume that 𝛾 = (𝛼, S𝛽) ∈ Sincomp 𝜃 . We distinguish between two cases.

• If 𝑏 = 0, then 𝛽 = 0 since (𝛼, 𝛽) ∈ 𝑉 𝜃 , hence 𝛽 ∈ Supp 𝑏 = {0} and S𝛽 = 0. The condition ‖𝛾‖ 2 = 1 therefore ensures that ‖(𝛼, 0)‖ 2 = 1 and 𝛼 ̸ = 0. Together with the condition 𝛼 ∈ (ker (P 𝜃 )) ⊥ , this implies that 𝛼 / ∈ ker (P 𝜃 ).

• If 𝑏 ̸ = 0, we use 𝛼 ∈ (ker (P 𝜃 )) ⊥ and distinguish again between two cases.

-If 𝛼 ̸ = 0, we deduce that 𝛼 / ∈ (ker (P 𝜃 )).

-If 𝛼 = 0, let us show that 𝛼 -𝛽 / ∈ ker (Q 𝜃 ). For the sake of contradiction, we assume that 𝛼 -𝛽 = -𝛽 ∈ ker (Q 𝜃 ), i.e. for all 𝑞 ∈ 𝒬,

Recalling that 𝛽 ∈ S -1 (Supp 𝑏 ) ⊆ ker P and recalling Proposition 3.3.6, we deduce that, for all 𝑞 ∈ 𝒬,

Let us show that (S𝛽) 𝜈 = 0 for every hidden neuron 𝜈 ∈ 𝐻. If 𝑏 𝜈 = 0, then (S𝛽) 𝜈 = 0 since S𝛽 ∈ Supp 𝑏 . If 𝑏 𝜈 ̸ = 0, since 𝜃 is admissible, there exists a path segment 𝑞 going from 𝜈 to the output neuron such that 𝑣 𝑞 (𝜃) ̸ = 0. Thus, since 𝑏 𝜈 𝑣 𝑞 (𝜃)(S𝛽) 𝜈 = 𝑏 𝑞 0 𝑣 𝑞 (𝜃)(S𝛽) 𝑞 0 = 0, we obtain (S𝛽) 𝜈 = 0 again.

To summarize, we have S𝛽 = 0. Since S is an isomorphism, we deduce that 𝛽 = 0. Since 𝛼 = 0, this contradicts the condition ‖𝛾‖ 2 = 1.

Sincomp 𝜃 is compact. By definition, Sincomp 𝜃 is a bounded subset of 𝑉 𝜃 . Let us show that it is closed. Let 𝛾 𝑛 = (𝛼 𝑛 , S𝛽 𝑛 ) ∈ Sincomp 𝜃 be a sequence that converges to some 𝛾 = (𝛼, S𝛽) ∈ Supp 𝜃 . For all 𝑛 ∈ N, 𝛾 𝑛 satisfies:

(2) 𝛼 𝑛 ∈ (ker P 𝜃 ) ⊥ .

(3) ‖𝛾 𝑛 ‖ 2 = 1.

As the set (ker P 𝜃 ) ⊥ is a finite-dimensional linear space, it is closed. Similarly the unit sphere of Supp 𝜃 is a closed set, hence properties (1) and ( 2) are also satisfied for 𝛾, showing that 𝛾 ∈ Sincomp 𝜃 . Hidden neurons are depicted as circles whereas input and output neurons as squares.

A.2.4 Illustrations on Particular Networks

We illustrate the main tools developed in Section 3.3 on two particular networks to showcase their usefulness.

Stem Network

We introduce a particular architecture 𝐺 defined in Figure A-1 that we call the Stem Network. We enumerate the connections as 𝐸 = (𝜇 → 𝜈, 𝜈 → 𝜂, 𝜂 → 𝜅), the only full path as 𝒫 = (𝜇 → 𝜈 → 𝜂 → 𝜅) and the partial paths as 𝒬 = (𝜈 → 𝜂 → 𝜅, 𝜂 → 𝜅). Using Definition 3.3.1, we compute P and Q as follows:

Thus both P and Q have full rank and

We value 𝐺 with 𝜃 = (𝑤, 𝑏), where 𝑤 = (𝑤 𝜇→𝜈 , 𝑤 𝜈→𝜂 , 𝑤 𝜂→𝜅 ) and 𝑏 = (𝑏 𝜈 , 𝑏 𝜂 ). We assume that 𝜃 is admissible (note that this implies that 𝑣 𝑝 (𝜃) ̸ = 0 for the only path 𝑝). In this particular case, this is equivalent to 𝑤 𝑒 ̸ = 0 for all 𝑒 ∈ 𝐸. Let us assume further that 𝑏 𝜈 ̸ = 0 and 𝑏 𝜂 ̸ = 0. Then, using the Notations,

Similarly, Supp 𝑏 = R 𝐻 ∖{0}. Using Propositions 3.3.6 and A.2.2, we deduce that

where

To compute 𝑏 ′ , we used the definition of S in Propositions 3.3.6. Let us now compute the set of 𝜃-compatible perturbations using Definition 3.3.3. First, using Definition 3.3.2, remark that P = P 𝜃 and Q = Q 𝜃 since 𝑣 𝑝 (𝜃) ̸ = 0 for for only full path 𝑝 ∈ 𝒫 and since 𝑏 𝜈 ̸ = 0 for all 𝜈 ∈ 𝐻.

Recalling that 𝛾 ∈ Supp 𝜃 and using Equation (A.102), we also have 0 = 𝛽 𝜇→𝜈 + 𝛽 𝜈→𝜂 + 𝛽 𝜂→𝜅 , hence 𝛽 𝜇→𝜈 = 0. Plugging these results into Equation (A.104), and re-stating Equation (A.103) here for clarity, we express 𝜃 ′ only in terms of 𝛼 as follows:

We now verify that the path-wise rescaling equivalence Definition 2.2.4 is satisfied.

For the only path 𝑝, using Equation A.109,

where we used Equation (A.99). Similarly, we have

We deduce that 𝜆 𝜈 = exp(-𝛼 𝜈→𝜂 -𝛼 𝜂→𝜅 ) = exp(𝛼 𝜇→𝜈 ) and 𝜆 𝜂 = exp(-𝛼 𝜂→𝜅 ). Then, we recover the neuron-wise rescaling equivalence. Indeed, 𝑤 𝜇→𝜈 and 𝑏 𝜈 are multiplied by 𝜆 𝜈 , 𝑤 𝜈→𝜂 is multiplied by 𝜆 𝜂 /𝜆 𝜈 = exp(-𝛼 𝜂→𝜅 -𝛼 𝜇→𝜈 ) = exp(𝛼 𝜈→𝜂 ), 𝑏 𝜂 is multiplied by 𝜆 𝜂 and 𝑤 𝜂→𝜅 is multiplied by 1/𝜆 𝜂 = exp(𝛼 𝜂→𝜅 ). Thus, this examples shows the duality between the weights and the biases to define rescaling-compatible perturbations, where the bridge between both spaces is implemented by S. We also use this technique to prove the Trajectory =⇒ Layer result Appendix A.1.1.

Sawtooth Network

We introduce an architecture 𝐺 in Figure A-1 called the Sawtooth Network 5 . We enumerate the connections

and the partial paths as

. Unlike with the previous example, we will not display all the calculations for the sake of concision. We compute

Thus both P and Q have full rank and

We value 𝐺 with 𝜃 = (𝑤, 𝑏). We assume that 𝜃 is admissible (note that this implies that 𝑣 𝑝 (𝜃) ̸ = 0 for both full paths). In this particular case, this is equivalent to 𝑤 𝑒 ̸ = 0 for all 𝑒 ∈ 𝐸. Let us assume further that 𝑏 𝜈 1 = 𝑏 𝜈 2 = 𝑏 𝜂 = 0. Then,

Similarly, Supp 𝑏 = {0}. Using Propositions 3.3.6 and A.2.2, we deduce that

Then, 𝛾 ∈ Supp 𝜃 writes 𝛾 = (𝛼, S𝛽) with (𝛼, 𝛽) ∈ 𝑉 𝜃 , hence 𝛾 = (𝛼, 0) using Equation A.118. The perturbed version of 𝜃 writes

Let us now compute the set of 𝜃-compatible perturbations using Definition 3.3.3. First, using Definition 3.3.2, remark that P = P 𝜃 since 𝑣 𝑝 (𝜃) ̸ = 0 for for both full paths. Then, using Definition 3.3.3, 𝛾 is 𝜃-compatible if, and only if, 𝛼 ∈ ker (P 𝜃 ) . This allows to verify that the path-wise rescaling equivalence Definition 2.2.4 is satisfied as in the previous example. Using the description of ker P, for the path

We deduce as in the previous example:

𝜆 𝜂 = exp(-𝛼 𝜂→𝜅 ). Then, we recover the neuron-wise rescaling equivalence.

A.3 Locally Identifiable Parameterizations

Since 𝜃 is restricted locally non-identifiable, there exists

To conclude, we show that 𝜏 can be chosen arbitrarily small. By the definition of 𝛼 ⊥ as an orthogonal projection onto (ker P 𝜃 ) ⊥ , we have ‖𝛼 ⊥ ‖ 2 ≤ ‖𝛼‖ 2 . Moreover, let us denote ‖𝑓 ‖ = ‖𝑓 ‖ 2,2 the operator norm of any linear map 𝑓 : 𝐸 → 𝐹 , defined as

Recalling that:

Recalling that ‖(𝛼, 𝛽)‖ 2 ≤ 𝜀 with Equation (A.121), we deduce that ‖𝛼‖ 2 ≤ 𝜀 and ‖𝛽‖ 2 ≤ 𝜀. Thus,

Combining the above estimates, we get that 𝜏 ≤ 𝜀(2 + ‖S -1 ‖‖T‖) can be chosen arbitrarily small by properly choosing 𝜀.

A.3.2 Sufficient Condition for Restricted Local Identifiability

where 𝜌 𝛾,𝑥 is defined in the proof of Proposition 3.4.3. Then, using Proposition 3.3.7,

Recalling that 𝛾 ∈ Sincomp 𝜃 (Definition 3.3.5), we have 𝛼 ∈ ker (P 𝜃 ) ⊥ and ‖𝛾‖ 2 = 1.

In particular, 𝛾 = (𝛼, 0) ̸ = 0, hence 𝛼 ̸ = 0 and P 𝜃 (𝛼) ̸ = 0. Moreover, since 𝜃 is admissible, we have 𝑣 𝑝 (𝜃) ̸ = 0 for all full path 𝑝. Hence, the vector 𝛿 ∈ R 𝒫 such that

Proof of Proposition 3.4.9

Proposition. Let 𝐺 be a one hidden layer architecture with scalar output valued with an admissible parameterization 𝜃. We further assume that there are no twin neurons, i.e. that 𝜃 is irreducible as defined in 3.2.4, Then, 𝒞 ⊥ 𝜃 = {0}.

Proof. Assume that there are no twin neurons. Let us show that 𝒞 ⊥ 𝜃 = {0}. Since 𝐺 is a one hidden layer architecture with scalar output, any full path 𝑝 = 𝜇 → 𝜈 → 𝜂 ∈ 𝒫 is given by an input neuron 𝜇 ∈ 𝑁 0 and a hidden neuron 𝜈 ∈ 𝑁 1 . Indeed, 𝜂 is the only output neuron. Using the Notations, recall that for any 𝜇 ∈ 𝑁 1 and 𝜈 ∈ 𝑁 2 , 𝑎 𝜇→𝜈→𝜂 (𝜃, 𝑥) = 𝑎 𝜈 (𝜃, 𝑥).

(

A.3.4 Current Limitations and Discussions

Proof of Proposition 3.4.10

Proposition. Assume that 𝜃 satisfies the following property

Then, the two following properties are equivalent:

(i) There exists 𝜀 > 0 such that, for all 𝜃 ′ ∈ 𝐵(𝜃, 𝜀) such that 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 , we have 𝑢(𝜃 ′ ) = 𝑢(𝜃).

(ii) There exists 𝜂 > 0 such that (Im(𝑢)

Proof. The implication (𝑖𝑖) =⇒ (𝑖) hold even without assuming (*). Assume that (𝑖𝑖) holds for some 𝜂 > 0. Since the function 𝑢 is polynomial, it is locally Lipschitz and we have

Let us now prove (𝑖) =⇒ (𝑖𝑖) using (*). Let 𝜀 > 0 satisfying (𝑖), that we re-write (𝑢(𝐵(𝜃, 𝜀)) -𝑢(𝜃)) ∩ 𝒞 ⊥ 𝜃 = {0}. Using (*), there exists 𝜂 > 0 such that (Im(𝑢) -𝑢(𝜃)) ∩ 𝐵 ∞ (0, 𝜂) ⊂ 𝑢(𝐵(𝜃, 𝜀)) -𝑢(𝜃). We conclude that

(2) For all 𝑧 ∈ Z𝜃 , 𝑐(𝜃 ′ , 𝑧) = 𝑐(𝜃, 𝑧)

Then the two following properties are equivalent:

(i) 𝜃 is locally identifiable.

(ii) There exists 𝜂 > 0 such that (Im(𝑢) -𝑢(𝜃)) ∩ 𝐵 ∞ (0, 𝜂) ∩ 𝒞 ⊥ 𝜃 = {0}. Proof. Let us show that (𝑖𝑖) =⇒ (𝑖). Since we assume (*), we leverage Proposition 3.4.10 to deduce from (𝑖𝑖) that there exists 𝜀 1 > 0 such that, for all 𝜃 ′ ∈ 𝐵(𝜃, 𝜀 1 ), if 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 , then 𝑢(𝜃 ′ ) = 𝑢(𝜃). Using Proposition 3.4.7, we deduce that 𝜃 is locally identifiable.

Let us show (𝑖) =⇒ (𝑖𝑖). Since 𝜃 is locally identifiable, there exists 𝜀 0 > 0 such that, for all 𝜃 ′ ∈ 𝐵(𝜃, 𝜀 > 0), 𝑅 𝜃 = 𝑅 𝜃 ′ implies that 𝑢(𝜃) = 𝑢(𝜃 ′ ).

Let 𝜀 ′ = min(𝜀, 𝜀 0 ) and 𝜃 ′ ∈ 𝐵(𝜃, 𝜀 ′ ) such that 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈ 𝒞 ⊥ 𝜃 . Let us show that 𝑢(𝜃) = 𝑢(𝜃 ′ ). We will then be able to conclude with Proposition 3.4.10 since 𝜃 satisfies (*).

For 𝑧 ∈, assumption (2) implies that 𝑐(𝜃 ′ , 𝑧) = 𝑐(𝜃, 𝑧). Since 𝑢(𝜃 ′ ) -𝑢(𝜃) ∈⊥ 𝐶 𝜃 , we have Let 𝜀 > 0. Consider 0 < 𝜂 ≤ min 𝑞∈𝒫∪𝒬 𝑢(𝜃) 𝑞 . For all 𝜃 ′ such that ‖𝑢(𝜃 ′ ) -𝑢(𝜃)‖ < 𝜂, since 𝑢(𝜃 ′ ) has all its coefficients that are strictly positive, we have

Hence, without loss of generality, we assume that all the coefficients of 𝜃 ′ are strictly positive. Then, we have .168) where log 𝜃 ′′ 𝑓 (𝜃) + 𝑔(𝜃 ′ ). We have 𝑢(𝜃 ′′ ) = 𝑢(𝜃 ′ ). Finally, let us show that if 𝜂 is small enough, we have ‖𝜃 ′′ -𝜃‖ < 𝜀. To do that, we combine the fact that log 𝜃 ′′ -log 𝜃 = 𝑔(𝜃 ′ ) -𝑔(𝜃) = L † (𝑢(𝜃 ′ ) -𝑢(𝜃)) with the continuity of the exponential function and of L † .

Appendix B Proofs and Supplementary Results for Equi-normalization B.1 Illustration of the Effect of Equi-normalization

We apply ENorm to one randomly initialized fully connected network with 20 intermediary layers. All the layers have a size 500 × 500 and are initialized following the Xavier scheme. The network has been artificially unbalanced as follows: all the weights of layer 6 are multiplied by a factor 1.2 and all the weights of layer 12 are multiplied by 0.8, see We next define the shift operator 𝑆 : R 𝑉 → R 𝐸 0 such that, for every 𝑥 ∈ R 𝑉 ,

and the padding operator 𝑃 as

We define the extended shift operator 𝑆 𝐻 = 𝑆 ∘ 𝑃 . We are now ready to define our change of variables. We define 𝜒 = 𝜓 ∘ 𝑆 𝐻 where

and observe that

Since ℎ is a 𝒞 ∞ diffeomorphism, its differential [𝐷ℎ](𝑑) is invertible for any 𝑑. It follows that [𝐷𝜙](𝑑) = 0 if, and only if, [𝐷𝜒](ℎ(𝑑)) = 0. As 𝜒 is the composition of a strictly convex function, 𝜓, and a linear injective function, 𝑆 𝐻 (proof after Step 3), it is strictly convex. Thus 𝜒 has at most one stationary point, which concludes this step. Note the similarities of this step with Proposition 3.3.6 in Chapter 3.

Step 3. We prove that the sequence 𝑑 (𝑟) converges. Step 1 implies that the sequence 𝑑 (𝑟) is bounded and has at least one cluster point, as 𝑓 is continuous on the compact 𝑋.

Step 2 implies that the sequence 𝑑 (𝑟) has at most one cluster point. We then use the fact that any bounded sequence with exactly one cluster point converges to conclude the proof.

𝑆 is injective. Let 𝑥 ∈ ker 𝑆 𝐻 . Let us show by induction on the hidden layer index 𝑘 that for every neuron 𝜈 at layer 𝑘, 𝑥 𝜈 = 0.

Appendix C Supplementary Results for iPQ and Quant-Noise

C.1 Quantization of Additional Architectures

ResNet-50. We explore the compression of ResNet-50, a standard architecture used Computer Vision. In Table C.1, we compare Quant-Noise to iPQ Compression from Chapter 5 and show that Quant-Noise provide consistent additional improvement.

C.2 Ablations

In this section, we examine the impact of the level of noise during training as well as the impact of approximating iPQ during training.

C.2.1 Impact of Noise Rate

We analyze the performance for various values of Quant-Noise in Figure C-1 on a Transformer for language modeling. For iPQ, performance is impacted by high rates of quantization noise. For example, a Transformer with the noise function 𝜙 proxy degrades with rate higher than 0.5, i.e., when half of the weights are passed through the noise function 𝜙 proxy . We hypothesize that for large quantities of noise, a larger effect of using proxy rather than the exact PQ noise is observed. For int8 quantization and its noise function, higher rates of noise are slightly worse but not as severe. A rate of 1 for int8 quantization is equivalent to the Quantization Aware Training of [START_REF] Krishnamoorthi | Quantizing deep convolutional networks for efficient inference: A whitepaper[END_REF]

C.3 Experimental Setting

We assess the effectiveness of Quant-Noise on competitive language and vision benchmarks. We consider Transformers for language modeling, RoBERTa for pre-training sentence representations, and EfficientNet for image classification. Our models are implemented in PyTorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF]. We use fairseq [START_REF] Ott | fairseq: A fast, extensible toolkit for sequence modeling[END_REF] for language modeling and pre-training for sentence representation tasks and Classy

Vision [START_REF] Adcock | Classy vision[END_REF] for EfficientNet.

Language Modeling. We experiment on the Wikitext-103 benchmark [START_REF] Merity | Pointer Sentinel Mixture Models[END_REF] that contains 100M tokens and a vocabulary of 260k words. We train a 16 layer Transformer following [START_REF] Baevski | Adaptive input representations for neural language modeling[END_REF] with a LayerDrop rate of 0.2 [START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF]. We report perplexity (PPL) on the test set.

Pre-Training of Sentence Representations. We pre-train the base BERT model [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF] on the BooksCorpus + Wiki dataset with a LayerDrop rate of 0.2. We finetune the pre-trained models on the MNLI task [START_REF] Williams | A broad-coverage challenge corpus for sentence understanding through inference[END_REF] from the GLUE Benchmark (Wang et al., 2019a) and report accuracy. We follow the parameters in Liu et al. (2019b) training and finetuning.

Image Classification. We train an EfficientNet-B3 model [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] on the ImageNet object classification benchmark [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. The EfficientNet-B3 of Classy Vision achieves a Top-1 accuracy of 81.5%, which is slightly below than the performance of 81.9% reported by [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF].

C.3.1 Training Details

Language Modeling To handle the large vocabulary of Wikitext-103, we follow (Dauphin et al., 2017) and [START_REF] Baevski | Adaptive input representations for neural language modeling[END_REF]. We use the adaptive softmax [START_REF] Grave | Efficient softmax approximation for gpus[END_REF] and adaptive input for computational efficiency. For both input and output embeddings, we use dimension size 1024 and three adaptive bands: 20K, 40K, and 200K. We use a cosine learning rate schedule [START_REF] Baevski | Adaptive input representations for neural language modeling[END_REF][START_REF] Loshchilov | Sgdr: Stochastic gradient descent with warm restarts[END_REF] and train with Nesterov's accelerated gradient [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF]. We set the momentum to 0.99 and renormalize gradients if the norm exceeds 0.1 [START_REF] Pascanu | How to construct deep recurrent neural networks[END_REF]. During training, we partition the data into blocks of contiguous tokens that ignore document boundaries. At test time, we respect sentence boundaries. We set LayerDrop to 0.2. We set Quant-Noise value to 0.05. During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise. During training, the block size is 8.

RoBERTa

The base architecture is a 12 layer model with embedding size 768 and FFN size 3072. We follow (Liu et al., 2019b) in using the subword tokenization scheme from [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], which uses bytes as subword units. This eliminates unknown tokens. We train with large batches of size 8192 and maintain this batch size using gradient accumulation. We do not use next sentence prediction [START_REF] Lample | Cross-lingual language model pretraining[END_REF]. We optimize with Adam with a polynomial decay learning rate schedule. We set LayerDrop to 0.2. We set Quant-Noise value to 0.1. We did not hyperparameter search to determine the optimal value of Quant-Noise as training RoBERTa is computationally intensive. During training time, the block size of Quant-Noise is 8.

During finetuning, we hyperparameter search over three learning rate options (1e-5, 2e-5, 3e-5) and batchsize (16 or 32 sentences). The other parameters are set following (Liu et al., 2019b). We do single task finetuning, meaning we only tune on the data provided for the given natural language understanding task. We do not perform ensembling. When finetuning models trained with LayerDrop, we apply LayerDrop and Quant-Noise during finetuning time as well.

EfficientNet We use the architecture of EfficientNet-B3 defined in Classy Vision [START_REF] Adcock | Classy vision[END_REF] and follow the default hyperparameters for training. We set Quant-Noise value to 0.1. During training time, we searched over the parameters (0.05, 0.1, 0.2) to determine the optimal value of Quant-Noise. During training time, the block size of Quant-Noise is set to 4 for all 1 × 1 convolutions, 9 for depth-wise 3 × 3 convolutions, 5 for depth-wise 5 × 5 convolutions and 4 for the classifier. For sharing, we shared weights between blocks 9-10, 11-12, 14-15, 16-17, 19-20-21, 22-23 and refer to blocks that share the same weights as a chunk. For LayerDrop, we drop

Model

MB PPL

Trans XL Large (Dai et al.,

C.3.2 Scalar Quantization Details

We emulate scalar quantization by quantizing the weights and the activations. The scales and zero points of activations are determined by doing a few forward passes ahead of the evaluation and then fixed. We use the Histogram method to compute 𝑠 and 𝑧, which aims at approximately minimizing the 𝐿 2 quantization error by adjusting 𝑠 and 𝑧. This scheme is a refinement of the MinMax scheme. Per channel quantization is also discussed in Table C.7.

C.3.3 iPQ Quantization Details

C.3.4 Details of Pruning and Layer Sharing

We apply the Every Other Layer strategy from [START_REF] Fan | Reducing transformer depth on demand with structured dropout[END_REF]. When combining layer sharing with pruning, we train models with shared layers and then prune chunks of shared layers. When sharing layers, the weights of adjacent layers are shared in chunks of two. For a concrete example, imagine we have a model with layers A, B, C, D, E, F, G, H. We share layers A and B, C and D, E and F, G and H. To prune, every other chunk would be pruned away, for example we could prune A, B, E, F.

C.4 Numerical Results for Graphical Diagrams

We (He et al., 2015a) 97.5 76.1 DenseNet-169 (Huang et al., 2018a) 53.4 76.2 EfficientNet-B0 [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] 20.2 77.3 MobileNet-v2 (Sandler et al., 2018a) 13.4 71.9 Shufflenet-v2 ×1 [START_REF] Ma | Shufflenet V2: practical guidelines for efficient CNN architecture design[END_REF] 8.7 69.4 HAQ 4 bits (Wang et al., 2018a) 12.4 76.2 iPQ ResNet-50 (Stock et al., 2019b)

C.5 Further Ablations

C.5.1 Impact of Quant-Noise for the Vision setup

We provide another study showing the impact of the proportion of elements on which to apply Quant-Noise in Table C.6.

C.5.2 Impact of the number of centroids

We quantize with 256 centroids which represents a balance between size and representation capacity. The effect of the number of centroids on performance and size is shown in Figure C-2 (a). Quantizing with more centroids improves perplexity -this parameter could be adjusted based on the practical storage constraints.

C.5.3 Effect of Initial Model Size

Large, overparameterized models are more easily compressed. In Figure C-3, we explore quantizing both shallower and skinnier models. For shallow models, the

Appendix D Supplementary Results for FaceGen D.1 Additional Comparative Results

D.1.1 Quality Evaluation: Ablation Studies

For evaluation, we assembled 28 videos of diverse persons in terms of gender, age, skin color from the validation set of VoxCeleb2 [START_REF] Joon | Voxceleb2: Deep speaker recognition[END_REF], and a similar set of 50 videos from the validation split of the DFDC dataset [START_REF] Dolhansky | The deepfake detection challenge (dfdc) dataset[END_REF].

We begin our analysis of the FOM by computing the quality of reconstruction without first order motion approximation and with/without adversarial training in Table D.1. While it is clear that the adversarial fine-tuning boosts the performance, we experiment without it in the remaining of our ablation study around this model to reduce training time for each model. Removing the first order approximation only slightly degrades the LPIPS but not the msVGG perceptual metric. Interestingly, the CSIM metric which is the one supposed to best reflect the identify preservation, is slightly increased by dropping this component. A second observation is that the fidelity of facial landmarks to the target video is negatively affected by this removal. Since the drop of performance induced by discarding first order motion approximation leads to important bandwidth savings and limited loss in performance, we conduct our experiments without it. We refer to this approach as the Motion Net approach. Next, we explore the replacement of the self-supervised landmarks of the Motion Net approach by off-the-shelf landmarks from a state-of-the art detector. Results appear in Table D.2. Note that the results presented in this table are obtained by our reimplementation of the MotionNet approach, and are slightly better than these of