Pierre Wolinski
email: pierre.wolinski@u-psud.fr

Guillaume Charpiat
email: guillaume.charpiat@inria.fr

Yann Ollivier

Interpreting a Penalty as the Influence of a Bayesian Prior

In machine learning, it is common to optimize the parameters of a probabilistic model, modulated by a somewhat ad hoc regularization term that penalizes some values of the parameters. Regularization terms appear naturally in Variational Inference (VI), a tractable way to approximate Bayesian posteriors: the loss to optimize contains a Kullback-Leibler divergence term between the approximate posterior and a Bayesian prior. We fully characterize which regularizers can arise this way, and provide a systematic way to compute the corresponding prior. This viewpoint also provides a prediction for useful values of the regularization factor in neural networks. We apply this framework to regularizers such as L 1 or group-Lasso.

Introduction

Adding a penalty term to a loss, in order to make the trained model fit some user-defined property, is very common in machine learning. For instance, penalties are used to improve generalization, prune neurons or reduce the rank of tensors of weights. Therefore, usual penalties are mostly empirical and user-defined, and integrated to the loss as follows:

L(w) = (w) + r(w), with w the vector of all parameters in the network, (w) the error term and r(w) the penalty term.

From a Bayesian point of view, optimizing such a loss L is equivalent to finding the Maximum A Posteriori (MAP) of the parameters w given the training data and a prior α ∝ exp(-r). Indeed, assuming that the loss is a log-likelihood loss, namely, (w) = -ln p w (D) with dataset D, then minimizing L is equivalent to minimizing L MAP (w) = -ln p w (D) -ln(α(w)). Thus, within the MAP framework, we can interpret the penalty term r as the influence of a prior α [START_REF] David | A practical bayesian framework for backpropagation networks[END_REF].

However, the MAP approximates the Bayesian posterior very roughly, by taking its maximum. Variational Inference (VI) provides a variational posterior distribution rather than a single value, hopefully representing the Bayesian posterior much better. VI looks for the best posterior approximation within a family β u (w) of approximate posteriors over w, parameterized by a vector u. For instance, the weights w may be drawn from a Gaussian distribution with mean u and fixed variance. The loss to be minimized over u is then: L VI (β u ) = -E w∼βu ln p w (D) + KL(β u α) = E w∼βu (w)

data fit term + KL(β u α) penalty term , (1)
and is also an upper bound on the Bayesian negative log-likelihood of the data [START_REF] Michael I Jordan | An introduction to variational methods for graphical models[END_REF]. Here p w (D) is the likelihood of the full dataset D given w, (w) = -ln p w (D) is the log-likelihood loss, and α is the Bayesian prior. The posteriors β u that minimize this loss will be concentrated around values of w that assign high probability to the data, while not diverging too much from the prior α. Thus, the KL divergence term can be seen as a penalty r(•) over the vector u.

Contributions and outline. We start from the following question: given some arbitrary penalty r(•), does it admit such an interpretation? Does there exist a prior α such that for all u, r(u) = KL(β u α) (up to an additive constant)? If so, is there a systematic way to compute such α? First, we provide a necessary and sufficient condition (Theorem 1) over the penalty r, that ensures the existence of a prior α such that VI with prior α reproduces the penalty r. The theorem comes with an explicit formula for α. We recover the MAP case as a degenerate case (Section 4.3).

Thus, we are able to determine whether a penalty r makes sense in a Bayesian framework and can be interpreted as the influence of a prior. We find this to be a strong constraint on r (Section 4.2). Here the regularizer r operates on the variational posterior β u ; for deterministic β this reduces to r directly acting on w, and we recover the traditional MAP correspondence in this case (Section 4.3).

Second (Section 5), we propose a heuristic to predict a priori useful values of the penalty factor λ to be put in front of a penalty r for neural networks, potentially bypassing the usual hyperparameter search on λ. Namely, we posit that the Bayesian prior α(w) corresponding to λr should reasonably match what is known for good a priori values of weights w in neural networks, namely, that the variance of weights under the prior α should match the Glorot initialization procedure [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. This usually provides a specific value of λ. Moreover, the penalty size gets automatically adjusted depending on the width of the various layers in the network.

We test this prediction for various penalties (Section 6), including group-Lasso [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF], for which per-layer adjustment of the penalty is needed [START_REF] Jose | Learning the number of neurons in deep networks[END_REF]. Experimentally, the predicted value of the regularization factors leads to reasonably good results without extensive hyperparameter search. Still, the optimal penalization factor is found to be systematically about 0.01 to 0.1 times our predicted value, showing that our heuristic provides a usable order of magnitude but not a perfect value, and suggesting that the Bayesian VI viewpoint may over-regularize.

Related Work

Interpretations of existing empirical deep learning methods in a Bayesian variational inference framework include, for instance, variational drop-out [START_REF] Durk P Kingma | Variational dropout and the local reparameterization trick[END_REF], a version of drop-out which fits the Bayesian framework. Further developments of variational drop-out have been made by [START_REF] Molchanov | Variational dropout sparsifies deep neural networks[END_REF] for weight pruning and by [START_REF] Louizos | Bayesian compression for deep learning[END_REF] for neuron pruning.

Closer to our present work, the links between a penalized loss and the Bayesian point of view have previously been mentioned by [START_REF] Bruno | Sparse coding with an overcomplete basis set: A strategy employed by v1?[END_REF] with a few approximations. [START_REF] David | A practical bayesian framework for backpropagation networks[END_REF] noted the equivalence of the penalized loss L(w) = (w) + r(w) and the MAP loss L MAP (w) = (w) -ln(α(w)) when (w) is the negative log-likelihood of the training dataset given the weights w, and with a prior α(w) ∝ exp(-r(w)). That is, finding the vector ŵ minimizing a loss L can be equivalent to finding the MAP estimator ŵMAP by minimizing the loss L MAP with a well-chosen prior distribution α.

However, the MAP framework is not completely satisfying from a Bayesian point of view: instead of returning a distribution over the weights, which contains information about their uncertainty, it returns the most reasonable value. In order to evaluate this uncertainty, [START_REF] David | Bayesian model comparison and backprop nets[END_REF] proposed a second-order approximation of the Bayesian posterior. In the process, [START_REF] David | Probable networks and plausible predictionsa review of practical bayesian methods for supervised neural networks[END_REF] also proposed a complete Bayesian framework and interpretation of neural networks. Still, this approximation of the Bayesian posterior is quite limited.

In the same period, [START_REF] Hinton | Keeping neural networks simple[END_REF] applied the Minimum Description Length (MDL) principle to neural networks. Then, [START_REF] David | Information theory, inference and learning algorithms[END_REF] made the link between the MDL principle and variational inference, and [START_REF] Graves | Practical variational inference for neural networks[END_REF] applied it to neural networks, allowing for variational approximations of the Bayesian posterior in a tractable way.

Variational Inference

We include here a reminder on variational inference for neural networks, following [START_REF] Graves | Practical variational inference for neural networks[END_REF].

From a Bayesian viewpoint, we describe the vector of weights w ∈ R N of a neural network as a random variable. Given a dataset D, we denote by p w (D) the probability given to D by the network with parameter w. For instance, with a dataset D = {(x 1 , y 1 ), • • • , (x n , y n )} of n input-output pairs and a model that outputs (log-)probabilities p w (y i |x i ) for the outputs, then ln p w (D) = n i=1 ln p w (y i |x i ) is the total log-likelihood of the data given the model. Given the dataset D, the posterior distribution over weights w is:

π D (w) = p w (D)α(w) P(D) , P(D) = w α(w)p w (D)
which is analytically intractable for multi-layer nonlinear neural networks. However, the posterior π D can be approximated by looking for probability distributions β that minimize the loss

L VI (β) = -E w∼β ln p w (D) + KL(β α), (2)
where KL(β α) = R N ln β(w) α(w) β(w) dw is the Kullback-Leibler divergence. Indeed, one has

L VI (β) = -ln P(D) + KL(β π D ), which is minimal when β = π D .
The first term in the loss (2) represents the error made over the dataset D: it is small if β is concentrated around good parameters w. The second term can be seen as a userdefined penalty over β that keeps it from diverging too much from the prior α. Moreover, for any distribution β, the quantity L VI (β) is a bound on the Bayesian log-likelihood of the data: L VI (β) ≥ -ln w α(w)p w (D) [START_REF] Michael I Jordan | An introduction to variational methods for graphical models[END_REF].

In variational inference, a parametric family B of probability distributions β is fixed, and one looks for the best approximation β * of the Bayesian posterior in B by minimizing L VI (β) in this family: the variational posterior. Importantly, for some families such as Gaussians with fixed variance, the gradient of L VI (β) can be computed if the gradients of ln p w (D) can be computed [START_REF] Graves | Practical variational inference for neural networks[END_REF], so that L VI (β) can be optimized by stochastic gradient descent. Thus, this is well-suited for models such as neural networks.

Thus, we consider a parametric family B = {β u : u ∈ R P } with parameter u, where each β u is a probability distribution over w ∈ R N . Then we learn the parameters u instead of the weights w. For instance, we can choose one of the following families of variational posteriors.

Example 1. The family of products of Gaussian distributions over w = (w 1 , • • • , w N ):

β u = β (µ1,σ 2 1 ,••• ,µ N ,σ 2 N ) = N (µ 1 , σ 2 1 ) ⊗ • • • ⊗ N (µ N , σ 2 N ).
In this case, the weights of the neural network are random and independently sampled from different Gaussian distributions N (µ k , σ 2 k ). Instead of learning them directly, the vector of parameters

u = (µ 1 , σ 2 1 , • • • , µ N , σ 2 N ) is learned to minimize L VI (β u ). Example 2.
The family of products of Dirac distributions over w = (w 1 , • • • , w N ):

β u = β (µ1,••• ,µ N ) = δ µ1 ⊗ • • • ⊗ δ µ N .
In this case, the weights are deterministic: w k and µ k are identical for all k.

4 Bayesian Interpretation of Penalties 4.1 When Can a Penalty Be Interpreted as a Prior?

In this section, we provide a necessary and sufficient condition that ensures that a penalty r(u) over the parameters of a variational posterior can be interpreted as a Kullback-Leibler divergence with respect to a prior α; namely, that

∃ K ∈ R, ∀u, r(u) = KL(β u α) + K
(the constant K does not affect optimization). In the process, we give a formula expressing α as a function of r(•).

In a nutshell, we place ourselves in the framework of variational inference: we assume the vector of weights w of the probabilistic model to be a random variable, drawn from a learned distribution β u parameterized by a vector u. For instance, the vector of weights w can be drawn from a multivariate normal distribution β µ,Σ ∼ N (µ, Σ).

We use some notions of distribution theory. We provide a reminder in Appendix C.

Notation. In order to approximate the posterior distribution of a vector w ∈ R N , we denote by (β µ,ν ) µ,ν the family of variational posteriors over w, parameterized by its mean µ and a vector of additional parameters ν. The basic example is a multivariate Gaussian distribution β µ,ν parameterized by its mean µ ∈ R N and its covariance matrix ν = Σ ∈ M N,N (R).

We say that the family (β µ,ν ) µ,ν of variational posteriors is translation-invariant if β µ,ν (θ) = β 0,ν (θ -µ) for all µ, ν, θ.

We denote by r(µ, ν) = r ν (µ) some penalty, to be applied to the distribution β µ,ν . We denote by F the Fourier transform, given by (Fϕ)(ξ) := R N ϕ(x)e -iξ•x dx for ϕ ∈ L 1 (R N ). This definition extends to the class of tempered distributions S (R N ) (see Appendix C).

In the sequel, we always restrict ourselves to priors α ∈ T (R N ) = {α s.t. ln(α) ∈ S (R N )}, i.e. log-tempered probability distributions, hence the condition ln(α) ∈ S (R N ) in the results below. This provides a reasonable behavior of α at infinity. Common probability distributions belong to this set. As a simple counter-example, one may take α(θ) ∝ exp(-exp |θ|), for which ln(α) / ∈ S (R).

Definition 1. We define the following distribution over R N :

A ν := -Ent(β 0,ν ) 1 -F -1 Fr ν F β0,ν , (3)
where β0,ν (θ) := β 0,ν (-θ) and 1 is the constant function equal to 1. We say that r fulfills ( ) ⇔ A ν does not depend on ν, i.e. A ν = A; A is a function such that exp(A) integrates to κ > 0.

Theorem 1. Let (β µ,ν ) µ,ν be a translation-invariant family of variational posteriors. Let r(µ, ν) = r ν (µ) be a penalty over β µ,ν . We assume that ∀ν, the probability distribution β 0,ν has finite entropy and lies in the Schwartz class S(R N ); that Fβ 0,ν is nonzero everywhere; and that Fr ν ∈ E (R N ), the class of distributions with compact support.

We are looking for probability distributions α such that:

∃ K ∈ R : ∀(µ, ν), r ν (µ) = KL(β µ,ν α) + K. ( 4
)
We have the following equivalence:

α is a solution to (4), with α ∈ T (R N ) ⇔ r fulfills ( ) and α = 1 κ exp(A),
where A is defined in Equation (3).

The proof is given in Appendix D. It is based on the resolution of a classical integral equation ( [START_REF] Andrei | Handbook of integral equations[END_REF], Section 10.3-1) adapted to the wider framework of distribution theory. This extension is necessary, since the Fourier transform of the widely-used L 2 penalty (r(x) = x 2 ) cannot be expressed as a function.

The preceding result holds under some technical assumptions. Even if some of the technical assumptions fail, the formula is still useful to compute a candidate prior α from a penalty r(•): apply Equation ( 3) on a penalty r to compute A ν , then check that Condition ( ) holds, define α = 1 κ exp(A), and finally compute KL(β µ,ν α) analytically and compare it to r. Remark 1. The assumption Fr ν ∈ E (R N ) includes the L 2 penalty and all L 2p penalties (for any positive integer p), but not the L 1 penalty. Indeed, For r 2 (x) = x 2 one has Fr 2 = -2πδ ∈ E (R). For r 2p (x) = x 2p , one has Fr 2p = (-1) p 2πδ (2p) ∈ E (R), but for r 1 (x) = |x|, (Fr 1 )(t) = 2t -2 / ∈ E (R). Still, for any penalty r ∈ S (R N ), it is always possible to find a sequence (r n ) n ∈ S (R N ) converging to r in S (R N ) such that Fr n ∈ E (R N ) for all n (see Appendix E).

Particular case: variational posteriors parameterized by their mean (ν = ∅). Theorem 1 provides a method to compute a prior from a penalty and a family of variational posteriors (β µ,ν ) µ,ν . Still, Eq. ( 3) returns a distribution A ν which may or may not satisfy the condition. Here we present a corollary which guarantees that the condition is satisfied; this holds under stricter conditions over the variational posteriors.

Corollary 1. Assume that the family of posterior distributions (β µ,ν ) µ,ν is only parameterized by their means, that is

β µ,ν = β µ and r ν (µ) = r(µ). Assume that β 0 ∈ S(R N ), Fr ∈ E (R N ),
Fβ 0 is nonzero everywhere and β 0 has a finite entropy. Assume that

F -1 [ F r F β0 ] is a function satisfying ∃ (a, b, k) ∈ R + * × R + * × R : ∀θ ∈ R N , F -1 Fr F β0 (θ) ≥ a|θ| b + k. ( 5
)
Then there exists κ > 0 such that:

α(θ) = 1 κ exp -Ent(β 0 ) -F -1 Fr F β0 (θ) = 1 κ e A(θ)
is a probability density satisfying r(µ) = KL(β µ α) up to a constant, and is the unique such probability density in T (R N ).

Remark 2. In many cases, condition (5) is not a limitation. For instance, with β µ ∼ N (µ, σ 2 ), where σ 2 is a constant, and with the penalty r the L 2 penalty, Condition (5) reads:

∃ (a, b, k) ∈ R + * × R + * × R such that θ 2 -σ 2 ≥ a|θ| b + k for all θ ∈ R N , which is satisfied.

Example 1: Gaussian Distributions with L 2 Penalty

Let us study the variational posteriors from Example 1: each vector of weights w ∈ R N is drawn from a Gaussian distribution

β u = N (µ 1 , σ 2 1 ) ⊗ • • • ⊗ N (µ N , σ 2 N ) with parameter u = (µ 1 , σ 2 1 , • • • , µ N , σ 2 N )
. We assume that each pair of variational parameters (µ k , σ k ) is penalized independently by some penalty r a,b depending on two real-valued functions a and b:

r a,b (µ k , σ 2 k ) = a(σ 2 k ) + b(σ 2 k )µ 2 k . (6)
The penalty over u is assumed to be the sum of the penalties r a,b (µ k , σ k ). Therefore, we study each pair (µ k , σ k ) independently and we omit the index k.

Corollary 2. If the penalty (6) above corresponds to a prior α, then α is Gaussian. More precisely, let α be a probability distribution in T (R), and assume that r a,b (µ, σ 2 ) = KL(β µ,σ 2 α), up to a constant. Then there exists σ 2 0 > 0 such that α = N (0, σ 2 0 ). Moreover, in that case, the penalty is, up to a constant:

r a,b (µ, σ 2 ) = 1 2 σ 2 + µ 2 σ 2 0 + ln σ 2 0 σ 2 -1 .
This corollary is an application of Theorem 1. We give the proof in Appendix G. Thus, the assumption that a penalty r arises from a variational interpretation is a strong constraint over r. Here the penalty r was initially parameterized by a pair of real functions (a, b), and is finally parameterized by a single number σ 2 0 .

Example 2: Deterministic Posteriors and the MAP

Another basic example is to use Dirac functions as variational posteriors (Example 2): β µ,ν = δ µ . Since δ 0 / ∈ S(R), the technical conditions of Theorem 1 are not satisfied. However, it is possible to apply Formula (3) and check that the resulting prior α is consistent with a chosen penalty r.

Applying Formula (3) yields

A = -Ent(δ 0 )1 -F -1 Fr Fδ 0 = -F -1 Fr 1 = -r.
Thus, if exp(-r) integrates to 0 < κ < ∞, then we can define α = 1 κ exp(-r). Then, we can check that indeed KL(δ µ α) = r(µ) up to a constant:

KL(δ µ α) = -Ent(δ 0 ) -δ µ , ln(α) = -ln α(µ) -Ent(δ 0 ) = r(µ) + ln κ -Ent(δ 0 ),
which confirms that the proposed prior α is consistent with the penalty r(•).

Thus, this formula recovers via variational inference the well-known penalty-prior equivalence in the MAP approximation, α VI (θ) ∝ exp(-r(θ)) ∝ α MAP (θ) [START_REF] David | A practical bayesian framework for backpropagation networks[END_REF].

However, this is somewhat formal: the entropy Ent(δ 0 ) of a Dirac function is technically undefined and is an "infinite constant". In practice, though, with a finite machine precision ε, a Dirac mass can be defined as a uniform distribution over an interval of size ε, and Ent(δ 0 ) becomes the finite constant ln ε.

Application to Neural Networks: Choosing the Penalty Factor

In this section, we compare the prior α arising from a penalty via Theorem 1, to reasonable weight priors for neural networks. In particular, we study how the prior varies when scaling the penalty by a factor λ, namely, r λ (•) = λr(•) for a reference penalty r. Requiring that α is comparable to standard priors for neural network weights provides a specific value of the regularization constant λ. Specifically, we compare the prior to the standard initialization of neuron weights [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]: the variance of weights sampled from the prior α should be approximately equal to the inverse of the number of incoming weights to a neuron. This constraint can be used to determine λ. In particular, this suggests different values of λ for different layers, depending on their size.

Possible advantages of being able to predict a good value for λ include avoiding a hyperparameter search, and better adjustment of the relative penalties of different layers or groups of neurons. For instance, one application of penalties in neural networks is to push neurons or convolutional filters towards zero, allowing for network pruning and reduced computational overhead. Penalties have been developed to remove entire neurons or filters, often based on the Lasso penalty: for instance, group-Lasso [START_REF] Scardapane | Group sparse regularization for deep neural networks[END_REF] and sparse group-Lasso [START_REF] Jose | Learning the number of neurons in deep networks[END_REF]. In the latter work, different penalties are used for different layers, with values of λ determined empirically. We will compare our predicted values of λ to the values used in these works.

Additive penalties and independent priors. Below, we focus on penalties that are expressed as sums over neurons or groups of neurons, such as L 2 or group-Lasso penalties. In the Bayesian setup, this corresponds to the additivity property of Kullback-Leibler divergence over products of distributions, KL(

β 1 ⊗ β 2 α 1 ⊗ α 2 ) = KL(β 1 α 1 ) + KL(β 2 α 2 )
. Thus, sums of penalties over different neurons or groups of neurons correspond to priors α and variational posteriors β that decompose as independent distributions over these neurons or groups of neurons.

A Reasonable Condition over the Prior α

Let us consider the variational inference framework applied to one weight w lij , the j-th weight of the i-th neuron in the l-th layer of a neural network. As a default distribution, one could expect α to be usable to initialize the weight w lij .

Therefore, we require α to satisfy the condition given by [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF] over the initialization procedure, which we call ( ):

α fulfills ( ) ⇔ E w lij ∼α [w lij ] = 0 and E w lij ∼α [w 2
lij ] = 1/P l where P l is the number of incoming weights in one neuron of the layer l. More generally, this condition can be written for the whole set of incoming weights to each neuron: denoting by w li the vector of all incoming weights of neuron i in layer l, one can define Condition ( ):

α fulfills ( ) ⇔ E w li ∼α [w li ] = 0 and E w li ∼α [ w li 2 ] = 1
which slightly extends ( ). Thus, if the prior α depends on some variable b, then Condition ( ) is reflected on b. For instance, in Example 1 (Section 4.2), ( ) is satisfied if and only if α σ 2 0 is N (0, 1/P l ). In the end, our suggested recipe for finding reasonable values for parameters b of a penalty r b is the following:

1. Follow the formulas in Thm. 1 to compute a prior α b such that r b (µ, ν) = KL(β µ,ν α b )+K.

2. write Condition ( ) or ( ) (depending on the case) for α b : this provides a constraint on b.

Examples

: L 2 , L 1 ,

and Group-Lasso Penalties

We now review some standard penalties, and apply this criterion to compute the penalty factor λ in front of each penalty.

Here we work with Dirac posterior distributions β µ = δ µ (Example 2 and Section 4.3). In that case, since each weight w lij is deterministically set to µ lij , we use w lij and µ lij indifferently. Thus, the penalty with penalty factor λ is

r λ (w) = r λ (µ) = λr(µ) = KL(δ µ α λ ) + K
for some reference penalty r. For each λ, the corresponding prior is α λ (θ) ∝ e -λr(θ) .

We will apply Condition ( ) to find a value for λ, for various penalties. In Table 1 we compare these values to usual ones.

We recall that P l in the number of incoming weights of a neuron in layer l.

Remark 3. We recall that r(w) is a full-dataset penalty (see Eq. 1), so that the actual perminibatch penalty for stochastic gradient is 1 n r(w) for individual samples or B n r(w) for minibatches of size B.

The results below apply equally to fully connected and to convolutional networks; in the latter case, "neuron" should read as "individual filter". L 2 penalty. We have r λ (w) = λw 2 , thus α λ (θ) ∼ N (0, 1/(2λ)). Then Condition ( ) is equivalent to λ = P l 2 . Thus, from a Bayesian viewpoint, when using a L 2 -penalty, each weight w of a neuron with P l incoming weights should be penalized by

r λ (w) = P l 2 w 2 . (7)
L 1 penalty. We have r λ (w) = λ|w|, thus α λ (θ) = λ 2 e -λ|θ| . Therefore, Condition ( ) is equivalent to λ = √ 2P l . As a consequence, when penalizing weight w in the l-th layer of a neural network with a L 1 -penalty, this weight w should be penalized with the term:

r λ (w) = 2P l |w|. ( 8
)
Standard Group-Lasso penalty. The group-Lasso jointly penalizes all the incoming weights of each neuron w l,i ∈ R P l . Thus, we consider a prior and a posterior that are probability distributions on R P l , and use Condition ( ). Denoting by w ∈ R P l the incoming weights of a neuron, we have

r λ (w) = λ w 2 . (9)
Then α λ (θ) = λ P l S P l -1 Γ(P l ) e -λ θ 2 , where Γ is Euler's Gamma function and S n-1 is the surface of the (n -1)-sphere. After computation of the variance, Condition ( ) is equivalent to λ = P l (P l + 1). As a consequence, when penalizing neuron w in the l-th layer of a neural network with a group-Lasso penalty, this neuron w should be penalized with the term: r λ (w) = P l (P l + 1) w 2 .

(

) 10
Table 1: How the regularization constant λ depends on the number of neurons n l and the number of parameters per neuron P l in layer l, both for our heuristics (λ Bayesian ) and for standard settings (λ usual ). For details on group-Lasso and reversed group-Lasso penalties, see Equations ( 9) and [START_REF] Louizos | Bayesian compression for deep learning[END_REF]. penalty L 2 L 1 group-Lasso rev. gr.-Lasso

λ Bayesian P l /2 √ 2P l P l (P l + 1) P l (n l + 1) λ usual 1 1 √ P l √ n l
This choice differs from [START_REF] Jose | Learning the number of neurons in deep networks[END_REF], who use λ ∝ √ P l , after an intuition proposed in [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF]. Their whole-network penalty is

L(w) = (w) + L l=1 λl (1 -γ) P l n l i=1 w l,i 2 + γ n l i=1 w l,i 1 , (11)
where γ ∈ [0, 1] is a fixed constant, L is the number of layers, n l is the number of neurons in the l-th layer, P l is the number of parameters in each neuron in the l-th layer, and w l,i is the set of weights of the i-th neuron of the l-th layer. The per-layer regularization constants λl are chosen empirically in [START_REF] Jose | Learning the number of neurons in deep networks[END_REF]. On the other hand, our Bayesian reasoning yields a scaling P l (P l + 1) instead of √ P l for the penalties w l,i 2 in [START_REF] Liu | Learning efficient convolutional networks through network slimming[END_REF].

Reversed Group-Lasso penalty. We recall that the standard group-Lasso penalty groups the weights of a layer by neurons, that is output features. It is also possible to group them by input features, namely, to group together the outgoing weights of each neuron. For a fully-connected neural network, the loss penalized by a "reversed" group-Lasso can be written:

L(w) = (w) + L l=1 λ l P l j=1 w l,•,j 2 , (12)
where w l,•,j ∈ R n l is the vector of weights of the l-th layer linked to the j-th input feature and n l is the number of neurons in the l-th layer. By computations similar to standard group-Lasso, ( ) is equivalent to λ = P l (n l + 1) where n l is the number of neurons in layer l, namely, r λ (w l,•,j ) = P l (n l + 1) w l,•,j 2 .

Experiments

Penalty terms are often used in the literature [START_REF] Scardapane | Group sparse regularization for deep neural networks[END_REF][START_REF] Jose | Learning the number of neurons in deep networks[END_REF][START_REF] Han | Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding[END_REF] in order to prune neural networks, that is, to make them sparse. Indeed, penalties such as L 1 or group-Lasso tend to push weights or entire groups of weights towards 0 [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF]. The efficiency of such methods can be measured in terms of final accuracy and number of remaining parameters.

We have chosen to compare within this context our heuristic to usual setups for the penalty factor, for different penalties. Our main criterion remains the final accuracy of the pruned network. According to the results given in Table 1, the global penalty should be decomposed into a sum of penalties over each layer:

r(w) = λ L l=1 λ l r(w l ),
where λ is a global penalty factor, L is the number of layers, r(w l ) is the penalty term due to the l-th layer and λ l is its corresponding penalty factor. According to our heuristic, each λ l should be set to λ Bayesian given in Table 1 and λ = λ Th = 1/n, where n is the size of the training dataset (by Remark 3). In the usual setup, each λ l is set to λ usual given in Table 1, while several values for λ are tested.

We test the quality of our heuristic in two steps. First, we only test the computed partial penalty factors λ l : they are fixed to λ Bayesian , while several values for λ are tested. Second, we test the full heuristic: each λ l is fixed to its corresponding λ Bayesian and λ = λ Th = 1/n. Experimental setup. We consider two neural networks: a version of VGG19 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] with one fully connected layer of size 512, and CVNN, which is a simple network with two convolutional layers (of respective sizes 100 and 200 with 5 × 5 patches), each followed by a ReLU and a 2 × 2 max-pool layer, and two fully-connected layers (of sizes 1000 and 200).

The training and pruning procedure is detailed in Appendix A and the full experimental procedure is detailed in Appendix B. The CIFAR-10 dataset is decomposed into three parts: a training set (42000 images), a validation set (8000 images), and a test set (10000 images). All reported accuracies are computed over the test set, which is never used during each run.

Results. We give the results in Figure 1. Each point gives the final accuracy and number of parameters of a neural network for a given setup, averaged over 3 runs. The red line and the blue line correspond respectively to the usual setup and our setup for per-layer penalty λ l , for various global factors λ. To check the quality of the heuristics over λ l , we should compare the maxima of the two lines in each graph. The theoretical value λ = λ Th is marked by the grey vertical line, so the blue point on the grey vertical line illustrates the performance of the joint theoretical values on both λ l and λ.

We report in Table 2 the values of relevant quantities: acc * usual , the best (over λ) accuracy of the usual per-layer scaling λ l ; acc * Bayesian , the best (over λ) accuracy of the Bayesian per-layer scaling λ l (we denote by λ * the optimal λ); and acc Bayesian , the accuracy of the Bayesian per-layer λ l together with the Bayesian predicted λ = λ Th . This would allow us to conclude:

if acc *

Bayesian > acc * usual , then we can conclude that our theoretical estimation for λ l is better that the usual ones; 2. moreover, we check whether acc Bayesian > acc * usual . If it is, then we can conclude that our theoretical estimation for λ l and the choice λ = λ Th are better than the usual ones; 3. finally, the closer the ratio λ Th /λ * is to 1, the closer we are to the Bayesian theoretical framework.

Regarding Point 1, our theoretical estimation for λ l leads to accuracies which remain close to accuracies obtained in the usual setup. Our setup leads to slightly better accuracies when training CVNN, and slightly worse for VGG19.

Regarding Points 2 and 3, the usual setup with optimized λ and our setup with Bayesian estimation for λ = λ Th perform similarly, though the second one usually performs slightly worse, due to a systematic mismatch between λ Th and λ * . Indeed, λ Th is always roughly 10 times greater than λ * .

CVNN VGG19

L 2 penalty Figure 1: Final performance and number of parameters for various penalties in function of the penalty factor λ. Red line: standard setup for the penalty; blue line: setup provided by the heuristic ( ) or ( ). Each bar corresponds to a final neural network: its abscissa is the penalty factor λ used for training it, its ordinate is its final accuracy, and its width corresponds to its final number of parameters. The vertical grey line is the value of the heuristic for λ.

Table 2: Comparison of the results. We show on the first row the best accuracy acc * usual obtained with the usual setup. Then we show its difference with the best accuracy acc * Bayesian obtained with our setup, its difference with the accuracy acc Bayesian obtained with our setup with the Bayesian theoretical value for λ, that is λ Th = n -1 , and the ratio between λ Th and the factor λ = λ * , that is the optimal value for λ. Among the results obtained with our methods, those highlighted in blue are better, and those highlighted in red are worse than in the usual setup. Discussion. This systematic overestimation of the penalty factor indicates that some phenomenon is not yet understood. We have two mutually compatible explanations. First, the choice λ = 1/n could be overestimated, as a strict Bayesian setup might be overcautious from the very beginning. Indeed, in other Bayesian approaches to neural networks, such observations have been made. For instance, when using stochastic Langevin dynamics to approximate the posterior, performance is better if the weight of the posterior is arbitrarily decreased by an additional factor n (see footnote 5 in [START_REF] Marceau-Caron | Natural langevin dynamics for neural networks[END_REF], and [START_REF] Li | Preconditioned stochastic gradient Langevin dynamics for deep neural networks[END_REF]).

Second, our choice for λ l , based on Glorot's initialization, could also be overestimated. Indeed, λ l only depends on P l (the number of parameters in each neuron in layer l), and not on the location of layer l in the network for instance, while the following observations suggest that the heuristic used to set λ l should take into account the architecture. We noticed that in most of the pruning experiments we made, the reached level of sparsity is much higher in the second half of the network (layers closer to the output). This is corroborated by [START_REF] Zhang | Are all layers created equal?[END_REF] which shows that the layers may behave differently from a training point of view, depending on the architecture of the network and their position in it. Notably, they have proven that, in a trained VGG, some of the last convolutional layers can be reset to their initial value without changing much the final accuracy, while this cannot be done for the first layers.

Conclusion

We have provided a theorem that bridges the gap between empirical penalties and Bayesian priors when learning the distribution of the parameters of a model. This way, various regularization techniques can be studied in the same Bayesian framework, and be seen as probability distributions. This unified point of view allowed us to take into account well known heuristics (as Glorot's initialization) in order to find reasonable values for hyperparameters of the penalty.

We have checked experimentally that our theoretical framework leads to reasonable results. However, we noticed a constant mismatch (about a factor 10) between our predicted penalty factor λ Th and the best one λ * . This fact raises interesting questions about a possible overcautiousness of the Bayesian framework and the possible impact of the architecture when penalizing layers, which we plan to investigate further.

C Reminder of Distribution Theory

In order to explain the main result, we recall some basic concepts of distribution theory. We use three functional spaces: the Schwartz class S(R N ), the space of tempered distributions S (R N ), and the space of distributions with compact support E (R N ).

Above all, we recall the definition of the space of distributions D (R N ). We denote by C ∞ (R N ) the space of infinitely derivable functions mapping R N to R, and by

C ∞ c (R N ) ⊂ C ∞ (R N
) the subspace of functions with compact support, that is ϕ ∈ C ∞ c (R N ) if, and only if:

ϕ ∈ C ∞ (R N ) and ∃K ⊂ R N compact s.t.: {x ∈ R N : ϕ(x) = 0} ⊆ K.
The set {x ∈ R N : ϕ(x) = 0} is also denoted by supp(ϕ).

Space of distributions D (R N ). The space of distributions D (R N ) is defined as the space of continuous linear forms over C ∞ c (R N ). For any distribution T ∈ D (R N ), we denote by T, φ the value of T at a given test function ϕ ∈ C ∞ c (R N ). More formally, T ∈ D (R N ) if, and only if, for all compact set K of R N , there exists p ∈ N and C > 0 such that:

∀ϕ ∈ C ∞ c (R N ) with supp(ϕ) ⊆ K, | T, ϕ | ≤ C sup |α|≤p ∂ α ϕ ∞ .
Distributions are easier to visualize in specific cases. For instance, if a function f : R N → R is integrable on every compact set K ∈ R N , then we can define a distribution T f by:

∀ϕ ∈ C ∞ c (R N ), T f , ϕ = f ϕ.
Therefore, distributions are often called "generalized functions", since most functions can be seen as distributions. In fact, by abuse of notation, f, ϕ stands for T f , ϕ .

Another classic example of distribution is the Dirac at zero δ, defined as follows:

∀ϕ ∈ C ∞ c (R N ), δ, ϕ = ϕ(0).
Schwartz class S(R N ). A function ϕ belongs to S(R N ) if, and only if ϕ ∈ C ∞ (R N ) and:

∀p ∈ N, ∃C p > 0 : sup |α|≤p,|β|≤p x α ∂ β ϕ(x) ∞ ≤ C p .
In few words, S(R N ) contains smooth and rapidly decreasing functions. For instance, any Gaussian density function belongs to S(R N ). We can easily define the Fourier transform on S(R N ). Let ϕ ∈ S(R N ):

(Fϕ)(ξ) = R N ϕ(x)e -iξx dx.
Thus, F -1 = (2π) -N F, where Fϕ = F φ and φ(x) = ϕ(-x).

Space of tempered distributions S (R N ). Let T ∈ D (R N ) be a distribution. T belongs to S (R N ) if, and only if, there exists p ∈ N and C > 0 such that:

∀ϕ ∈ C ∞ c (R N ), | T, ϕ | ≤ sup |α|≤p,|β|≤p x α ∂ β ϕ ∞ .
The Fourier transform is defined on S (R N ) by duality. For any T ∈ S (R N ), FT ∈ S (R N ) and is defined by: ∀ϕ ∈ S(R N ),

FT, ϕ = T, Fϕ .

Notably, this definition allows us to compute the Fourier transform of functions that do not lie in L 2 . This is very useful in the applications of Theorem 1, where we need the Fourier transform of f : x → x 2 , which is Ff = -2πδ ∈ S (R) (where δ is the second derivative of the Dirac, defined below).

Space of distributions with compact support E (R N ). Let T ∈ D (R N ). The support of T is defined by:

supp(T ) = R N \{x ∈ R N : ∃ω neighborhood of x s.t. T |ω = 0}.
Thus, T is said to have a compact support if, and only if, supp(T ) is contained into a compact subset of R N . As fundamental property of E (R N ), one should notice that E (R N ) ⊂ S (R N ). That is, the Fourier transform is defined on E (R N ).

For instance, the Dirac at zero δ and its derivatives δ (k) have support {0}, which is compact:

δ, ϕ = ϕ(0) δ (k) , ϕ = (-1) k ϕ (k) (0), for any test function ϕ ∈ C ∞ c (R N ).

D Proof of Theorem 1

Proof. The proof can be split into three parts:

1. according to Definition 1, we have:

A ν := -Ent(β 0,ν )1 -F -1 Fr ν F β0,ν ,
and we prove that, for all (µ, ν):

Ent(β 0,ν ) -A ν , β µ,ν = r ν (µ). (13)
2. we prove that, if exp •A ν is a function integrating to κ > 0, then there exists K ∈ R such that:

∀(µ, ν), r(µ, ν) = KL(β µ,ν α ν ) + K, (14)
where α ν = 1 κ exp •A ν . At that point, if Condition ( ) is satisfied, then r can be written as a KL-divergence with respect to a prior. Thus, Condition ( ) is a sufficient condition to ensure the existence of a solution to Equation (4).

3. we prove that, if α a solution to Equation (4), then ln •α is equal to A ν (up to a constant) and A ν satisfies Condition ( ).

In the proof, and notably in Appendix D.3, we need the following proposition.

Proposition 3. Let T ∈ S (R N ) and ϕ ∈ S(R N ). Then T * ϕ ∈ S (R N ) and:

F [T * ϕ] = (FT ) • (Fϕ).
Proof. This statement is directly given in the proof of Theorem 3.18 [START_REF] Elias | Introduction to Fourier analysis on Euclidean spaces (PMS-32)[END_REF].

D.1 Proof of Equation (13)

The distribution A ν is defined by:

A ν = -Ent(β 0,ν )1 -F -1 Fr ν F β0,ν . Since 1 ∈ S (R N ) and F rν F β0,ν ∈ S (R N ), then A ν ∈ S (R N
) by stability of S by Fourier transform ([6], Lemma 7.1.3).

We compute:

-

Ent(β 0,ν ) -A ν , β µ,ν = -Ent(β 0,ν ) + Ent(β 0,ν )1 + F -1 Fr ν F β0,ν , β µ,ν = F -1 Fr ν F β0,ν , β µ,ν , since β µ,ν = 1.
By definition of the Fourier transform over S (R N ) ([6], Definition 7.1.9), we have:

-Ent(β 0,ν ) -A ν , β µ,ν = Fr ν F β0,ν , F -1 β µ,ν . (15)
We compute F -1 β µ,ν :

(F -1 β µ,ν )(ξ) = (2π) -N e iξ• , β µ,ν (•) = (2π) -N e iξ• , β 0,ν (• -µ) = (2π) -N e iξ• , β0,ν (µ -•) ,
using the assumptions over the family (β µ,ν ) µ,ν . Thus:

(F -1 β µ,ν )(ξ) = (2π) -N e iξ(µ-•) , β0,ν (•) = (2π) -N e iξµ e -iξ• , β0,ν (•) = (2π) -N e iξµ (F β0,ν )(ξ).
injecting the result into Equation ( 15), we have:

-Ent(β 0,ν ) -A ν , β µ,ν = (2π) -N Fr ν F β0,ν , e iµ• (F β0,ν )(•) = (2π) -N Fr ν , e iµ• F β0,ν F β0,ν = (2π) -N Fr ν , e iµ• .
Since Fr ν ∈ E (R N ), we can apply Theorem 7.1.14 [START_REF] Hörmander | The Analysis of Linear Partial Differential Operators I[END_REF]:

(2π) -N Fr ν , e iµ• = (F -1 Fr ν )(µ) = r ν (µ),
which achieves the proof of Equation ( 13).

D.2 Proof of Equation (14)

Now, we assume that exp •A ν is a function integrating to κ > 0. We define α ν :

α ν = 1 κ exp •A ν ,
which is non-negative and integrates to 1. Thus, α ν is a density of probability. We compute the KL-divergence between β µ,ν and α ν :

KL(β µ,ν α ν ) = R N ln β µ,ν (θ) α ν (θ) β µ,ν (θ) dθ = -Ent(β µ,ν ) -ln •α ν , β µ,ν = -Ent(β µ,ν ) --ln(κ)1 + A ν , β µ,ν = -Ent(β µ,ν ) + ln(κ) 1, β µ,ν -A ν , β µ,ν = r ν (µ) + ln κ,
since β µ,ν integrates to 1 and Ent(β µ,ν ) does not depend on µ. Therefore, using Equation (13):

KL(β µ,ν α ν ) = r(µ, ν) + ln κ.
Moreover, if A ν does not depend on ν, then:

KL(β µ,ν α) = r(µ, ν) + ln κ.
Therefore, Condition ( ) is a sufficient condition to ensure the existence of a solution α to Equation (4).

D.3 Uniqueness of the Solution

We assume that α is a probability distribution in T (R N ) and:

r ν (µ) = KL(β µ,ν α) + K,
where K ∈ R is a Thus:

r ν (µ) = R N ln β µ,ν (θ) α(θ) β µ,ν (θ) dθ + K = -Ent(β µ,ν ) - R N [ln(α(θ)) -K] β µ,ν (θ) dθ = -Ent(β µ,ν ) - R N Â(θ)β µ,ν (θ) dθ (where Â(θ) := ln(α(θ)) -K) = -Ent(β µ,ν ) - R N Â(θ)β 0,ν (θ -µ) dθ = -Ent(β µ,ν ) - R N Â(θ) β0,ν (µ -θ) dθ = -Ent(β 0,ν ) -( Â * β0,ν )(µ),
since the convolution between  ∈ S (R N ) and β0,ν ∈ S(R N ) is well-defined ([23], Theorem 3.13), and Ent(β µ,ν ) = Ent(β 0,ν ). Then, we can apply the Fourier transform:

Fr ν = -2πEnt(β 0,ν )δ -F( Â * β0,ν ) = -2πEnt(β 0,ν )δ -(F Â) • (F β0,ν ),
by applying Proposition 3. Since Fβ 0,ν is supposed to be nonzero everywhere, then:

F Â = -2πEnt(β 0,ν )δ -Fr ν F β0,ν .
From now, we just have to compute the inverse Fourier transform of F Â to get Â: Â = F -1 -2πEnt(β 0,ν )δ -Fr ν F β0,ν = -F -1 2πEnt(β 0,ν )δ F β0,ν -F -1 Fr ν F β0,ν = -2πEnt(β 0,ν )F -1 δ F β0,ν -F -1 Fr ν F β0,ν .

We compute the first term, which is a tempered distribution. For all ϕ ∈ S(R N ), we have:

δ F β0,ν , ϕ = δ, ϕ F β0,ν = ϕ(0) (F β0,ν )(0)
= ϕ(0), since (F β0,ν )(0) is equal to β0,ν = β 0,ν = 1. Thus, δ F β0,ν = δ.

G Proof of Corollary 2

We to apply Theorem 1 in this case. We assume that α is a solution to Equation (4) with r a,b (µ k , σ 2 k ) = a(σ 2 k ) + b(σ 2 k )µ 2 k , such that α ∈ T (R). That is, for some constant K: r a,b (µ k , σ 2 k ) = KL(β µ k ,σ 2 k α) + K. We check the hypotheses:

• for all (µ, σ 2 ) ∈ R × R + , β µ,σ 2 (•) = β 0,σ 2 (• -µ) and β 0,σ 2 = β0,σ 2 ;

• β µ,σ 2 ∈ S(R) and Fβ µ,σ 2 is nonzero everywhere;

• Fr a,b,σ 2 = 2π a(σ 2 )δ -b(σ 2 )δ ∈ E (R).

Then, we apply Theorem 1, which ensures that r fulfills ( ), that is A a,b,σ 2 does not depend on σ 2 . First, we compute A a,b,σ 2 by using its definition, given in Equation [START_REF] Graves | Practical variational inference for neural networks[END_REF]. A first calculus (see Appendix G.1) leads to:

Fr a,b,σ 2 Fβ 0,σ 2 = 2π a(σ 2 ) -σ 2 b(σ 2 ) δ -b(σ 2 )δ , (16)
and, according to the calculus made in Appendix G.2, we have:

A a,b,σ 2 (θ) = -ln(2πeσ 2 ) + -a(σ 2 ) + σ 2 b(σ 2 ) -b(σ 2 )θ 2 . (17)
Second, we prove in Appendix G.3 that A a,b,σ 2 does not depend on σ 2 if, and only if:

a(σ 2 ) = σ 2 b 0 -a 0 - ln(2πeσ 2 ) 2 (18) b(σ 2 ) = b 0 , (19)
where a 0 and b 0 are real constants. Thus, the loss r a,b takes the following form:

Proof of Equation (16)

We compute F r a,b,σ 2 F β 0,σ 2 . We recall that Fr a,b,σ 2 = 2π a(σ 2 )δ -b(σ 2 )δ . Thus we have, for each ϕ ∈ S(R):

r.

  a,b (µ, σ 2 ) = σ 2 b 0 -a 0 -ln(2πeσ 2 ) 2 + b 0 µ 2 + K. In that case, A a,b,σ 2 (θ) = A a,b (θ) = a 0 -b 0 θ 2 . Thus α a,b = 1 κ exp •A a,b is a probability distribution, where: κ = e a0 2π 2b0Thus, α a,b can be seen as the density of the Gaussian distribution N (0, 1 2b0 ). Therefore α a,b is Gaussian and the penalty is necessarily the Kullback-Leibler divergence between two Gaussian distributions β µ,σ 2 ∼ N (µ, σ 2 ) and α a,b = α σ 2 0 ∼ N (0,

Fr a,b,σ 2 x 2 2 xx exp σ 2 x 2 2 xx 2 2 xF r a,b,σ 2 F 2 Fβ 0,σ 2 F r a,bσ 2 F β 0,σ 2 = 2 G. 3

 22222222223 Fβ 0,σ 2 , ϕ = 2π a(σ 2 )δ -b(σ 2 )δ Fβ 0,σ 2 , ϕ = 2π a(σ 2 )δ -b(σ 2 )δ , ϕ Fβ 0,σ 2 = 2πa(σ 2 )ϕ(0) -2πb(σ 2 ) δ , ϕ(x) exp σ 2 = 2πa(σ 2 )ϕ(0) + 2πb(σ 2 ) δ , ϕ (x) + ϕ(x)σ 2 = 2πa(σ 2 )ϕ(0) -2πb(σ 2 ) δ, ϕ (x) + 2ϕ (x)σ 2 x + ϕ(x)σ 2 (1 + σ 2 x 2 ) exp σ 2 = 2π a(σ 2 )ϕ(0) -b(σ 2 ) ϕ (0) + σ 2 ϕ(0) Thus, β 0,σ 2 = 2π a(σ 2 ) -σ 2 b(σ 2 ) δ -b(σ 2 )δ .G.2 Proof of Equation (17)Theorem 1 defines the following distribution:A a,b,σ 2 = -Ent(β 0,σ 2 )1 -F -1 Fr a,b,σWe have:Ent(β 0,σ 2 ) = 1 2 ln(2πeσ 2 ) 2π a(σ 2 ) -σ 2 b(σ 2 ) δ -b(σ 2 )δ Besides, 2πF -1 δ = 1 and 2π(F -1 δ )(θ) = -θ 2 .Thus:A a,b,σ 2 (θ) = -1 2 ln(2πeσ 2 ) -a(σ 2 ) -σ 2 b(σ 2 ) + b(σ 2 )θ22 ) + -a(σ 2 ) + σ 2 b(σ 2 ) -b(σ 2 )θ Proof of Conditions (18) and (19)

2 .

 2 2 ) + -a(σ 2 ) + σ 2 b(σ 2 ) -b(σ 2 )θThe polynomial A a,b,σ 2 does depend on σ 2 if, and only if, its coefficients do not depend on σ 2 . That is:b(σ 2 ) = b 0 -1 2 ln(2πeσ 2 ) -a(σ 2 ) + σ 2 b(σ 2 ) = a 0 , that is: b(σ 2 ) = b 0 a(σ 2 ) = σ 2 b 0 -a 0 -ln(2πeσ 2 ) 2.

A Training and Pruning: Details

Since pruning neurons causes an accuracy drop, we divided the learning into two phases: learning and pruning phase; fine-tuning phase. This trick is widely used in pruning literature [START_REF] Li | Pruning filters for efficient convnets[END_REF][START_REF] Scardapane | Group sparse regularization for deep neural networks[END_REF][START_REF] Liu | Learning efficient convolutional networks through network slimming[END_REF], in order to achieve better performance.

We define a pruning criterion based on the "norm" of each neuron. The tested penalties, i.e. L 2 penalty, L 1 penalty, group-Lasso penalty, and reversed group-Lasso penalty, can be separated into two categories: penalization of the output features of each layer (L 2 , L 1 , group-Lasso) and penalization of input features of each layer (reversed group-Lasso).

Pruning with L 2 , L 1 and group-Lasso penalties. For each neuron w l,i , we check whether:

If so, then the neuron is pruned.

Pruning with reversed group-Lasso penalty. For each vector w l,•,j of the j-th input weights of the neurons in layer l, we check whether:

If so, then the j-th neuron in layer l -1 is pruned, because its output is almost not used in layer l.

Pruning and training phase. The penalty is applied and, after each training epoch, pruning is performed over all layers. This phase ends when the number of neurons and the best validation accuracy have not improved for 50 epochs.

Fine-tuning phase. The penalty is removed and the learning rate is decreased by a factor 10 each time the validation accuracy has not improved for 50 epochs, up to 2 times. The third time, training is stopped. No pruning is performed during this phase.

B Experimental Procedure

For each combination of neural network (VGG19 or CVNN) and penalty (L 2 , L 1 , group-Lasso, or reversed group Lasso), we have tested two setups: our Bayesian heuristic for the penalty factor λ l of each layer l, and the usual setup for them (as described in Table 1). For each setup, we have planned to plot the final accuracy and final number of parameters in function of the global penalty factor λ. We have proceeded as follows:

1. we define a set Λ of λ we want to use into our experiments. Typically, we have chosen Λ = (1/n) • {10 -2.5 , 10 -2 , 10 -1.5 , 10 -1 , 10 -0.5 , 10 0 , 10 0.5 } in our setup (where n is the size of the training set), and Λ = {10 -6 , 10 -5.5 , 10 -5 , 10 -4.5 , 10 -4 , 10 -3.5 , 10 -3 , 10 -2.5 } in the usual setup;

2. for each λ ∈ Λ, we test several learning rates η ∈ {10 -2 , 10 -3 , 10 -4 }, and we select the learning rate η λ which led to the best accuracy;

3. for each λ ∈ Λ, we run 2 more experiments with the selected learning rate η λ . Thus, we are able to average our results over 3 runs.

Recalling that Â(θ) := ln(α(θ)) -nK, we have:

thus A ν does not depend on ν, from which we deduce that r fulfills Condition ( ) and α ∝ exp •A.

E Note on Remark 1

We show that, for all r ∈ S (R N ), there exists a sequence (r n ) n ∈ S (R N ) converging to r in S (R N ) such that Fr n ∈ E (R N ) for all n. For all r ∈ S (R N ), Fr ∈ S (R N ). Let us build the sequence of distributions :

The sequence (q n ) n ∈ E (R N ) converges to Fr in S (R N ). Since F -1 is continuous, we have:

Therefore, we can pose r n = F -1 q n , which is appropriate.

F Proof of Corollary 1

Proof. We apply Formula (3):

Assuming there exists a > 0, b > 0 and k ∈ R such that: Since exp -a|θ| b + k dθ converges, then exp(A(θ)) dθ converges. Thus, according to Theorem 1, there exists κ > 0 such that α = 1 κ exp •A verifies: KL(β µ α) = r(µ), up to a constant.


Jean-Baptiste Doré
email: jean-baptiste.dore@cea.fr

Didier Belot
email: didier.belot@cea.fr

Eric Mercier
email: eric.mercier@cea.fr

Simon Bicaïs

Grégory Gougeon

Yoann Corre
email: ycorre@siradel.com

Benoit Miscopein

Dimitri Kténas

Emilio Calvanese Strinati

Technology Roadmap for Beyond 5G Wireless Connectivity in D-band

Keywords: Millimeter wave communications, Integrated circuit technology, Radiofrequency integrated circuits

Wireless communication in millimeter wave bands, namely above 20 GHz and up to 300 GHz is foreseen as a key enabler technology for the next generation of wireless systems. The huge available bandwidth is contemplated to achieve high data rate wireless communications, and hence, to fulfill the requirements of future wireless networks. Several Beyond 5G applications are considered for these systems: high capacity backhaul, enhanced hot-spot kiosk as well as short-range Deviceto-Device communications. In this paper we propose to discuss the trade-offs between scenario requirements and current silicon technologies limits to draw a technology roadmap for the next generation of wireless connectivity in D-band.

I. INTRODUCTION

Given their large unused spectrum, the sub-TeraHertz (sub-THz) frequency bands between 90 and 300 GHz are potential candidates to achieve high-data rate wireless communications and hence to fulfill the requirements of future Beyond 5G (B5G) [1]. Considered applications include high capacity back-haul (100+ Gbps data rates) and enhanced hot-spot kiosk [START_REF] Hamada | Millimeter-wave InP Device Technologies for Ultrahigh Speed Wireless Communications toward Beyond 5G[END_REF] [START_REF] Doré | Above-90GHz Spectrum and Single-Carrier Waveform as Enablers for Efficient Tbit/s Wireless Communications[END_REF]. Moreover, short range or Device-to-Device (D2D) communications could be deployed in sub-THz as this new spectrum would significantly increase transmission rates. As an example, an aggregated bandwidth of 58.6 GHz was identified as possibly available for terrestrial radio-communications between 90 and 200 GHz. However before making this possible, many challenges need to be addressed. First of all, as the free-space propagation losses increase with the square of the frequency they have to be compensated by using highgain antennas, which entails severe constraints on antenna directivity and alignment. Besides, the connectivity link is mainly restricted to line-of-sight (LoS) or quasi-LoS situations, or requires a strong reflection. The design of antenna with electronically controlled steerable beam remains an open issue. Secondly, semiconductor technologies have to be chosen to meet expected performance at cost effective targets, including energy efficiency ones. The higher the frequency, the higher the price to pay [START_REF] Hamada | Millimeter-wave InP Device Technologies for Ultrahigh Speed Wireless Communications toward Beyond 5G[END_REF] [4] [START_REF] Rodríguez-Vázquez | A 16-QAM 100-Gb/s[END_REF] [START_REF] Dan | A Superheterodyne 300 GHz Wireless Link for Ultra-Fast Terahertz Communication Systems[END_REF] .

in addition to challenges on analog/digital conversion, new physical (PHY) layer algorithms are required on the digital side to achieve high data-rate as traditional coherent schemes

The research leading to these results received funding from the French National Research Agency (ANR-17-CE25-0013) within the frame of the project BRAVE.

do not consider the specific radiofrequency (RF) impairments related to the sub-THz bands. In particular, sub-THz systems could suffer from strong phase noise (PN) due to the poor performance of high-frequency oscillators [START_REF] Bicaïs | Phase Noise Model Selection for Sub-THz Communications[END_REF]. Therefore the study of new digital transmission schemes, optimized to mitigate such impacts, like PN, is essential to guarantee good performance [START_REF] Bicaïs | Optimized Single Carrier Transceiver for Future Sub-TeraHertz Applications[END_REF]. It should be noticed that an alternative PHY layer has been defined by the IEEE 802.15-3d [START_REF]IEEE Standard for High Data Rate Wireless Multi-Media Networks-Amendment 2: 100 Gb/s Wireless Switched Point-to-Point Physical Layer[END_REF] at the lower sub-THz frequency range between 252 GHz and 325 GHz for switched Point-to-Point links. Many others solutions are currently under investigation to reach a trade-off between throughput, power consumption and complexity -see [START_REF] Saad | Single carrier with index modulation for low power terabit systems[END_REF] as an example.

The contributions of this paper are the following. First, we review some potential scenarios in which the use of sub-THz bands could give new opportunities. Then we analyze three applications by proposing dedicated PHY layers and link budgets based on propagation environment derived from a sub-THz ray-based model. Eventually, applications and requirements are cross-correlated with intrinsic silicon and III-V technologies features. This enables us to identify a technology roadmap fitting with next generation of wireless connectivity devices.

II. CONTEMPLATED SCENARIOS A. System model

For mmWave bands, system models have to be reconsidered w.r.t sub-6GHz band ones. The LoS direct-path is often the most dominant component. The propagation channel properties affect the link-level performance, therefore the service coverage. A fine characterization is required for realistic evaluation of new sub-THz behavior. Only few channel sounding campaigns, complex and costly, have been published yet [START_REF] Pometcu | An Indoor Channel Model for High Datarate Communications in D-band[END_REF] [START_REF] Xing | Propagation Measurement System and Approach at 140 GHz-Moving to 6G and Above 100 GHz[END_REF]. They are complemented by convenient numerical prediction, and produce on-demand channel models. In this work we use a model based on ray-tracing, extended up to the sub-THz frequencies [START_REF] Gougeon | Ray-based Deterministic Channel Modelling for sub-THz Band[END_REF]. This Volcano tool benefits from a LiDAR point cloud or detailed 3D representations to get a realistic prediction of the blockages and losses due to trees and street or indoor furnitures. The ray-tracing tool then predicts the propagation of both in-street back-haul or indoor mall scenario.

Regarding the electronic systems themselves, a critical impact from RF impairments, such as non-linearity, IQ imbal-ance and PN are expected. RF front-end analog blocks nonlinearities increase challenges in both modeling of circuits and anticipating the compensation measures. Power Amplifier (PA) efficiency and achievable output power are important targets to meet. In addition, the digitalization of wide bandwidths (BW) with power and integration constraints is still an open research topic. Last, PN, described by an uncorrelated Gaussian process for wide BW causes significant degradation in the communication performance.

Finding the optimum modulation scheme with regards to PN specifications has been largely investigated in the literature and remains a crucial topic to fulfil the requirement of future wireless technologies. Two different approaches can be proposed. First, envelop modulation, allowing non-coherent demodulation, could be considered. The information is carried by the signal amplitude and hence is inherently robust to severe phase impairments. Demodulation performance are far from the theoretical capacity but simple and energy optimized transceivers can be designed. In this case, maximizing the spectral efficiency per unit of energy is the optimization criterion. Second, when it comes to high spectral efficiency applications with band limited constraint, waveforms based on coherent modulations should be optimized to be robust to PN. As a result, we have proposed in recent work the Polar Quadrature Amplitude Modulation (P-QAM) [START_REF] Bicaïs | Optimized Single Carrier Transceiver for Future Sub-TeraHertz Applications[END_REF]. This modulation is defined as a rectangular lattice in the amplitudephase domain. The P-QAM balances coded information in the amplitude and phase components to meet constraints on PN robustness and Peak-to-Average Power Ratio (PAPR). Next sections of the paper will consider, for coherent demodulation, the P-QAM combined with 5GNR LDPC (Low Density Parity Check) FEC (Forward Error Correction) scheme for performance assessments.

B. Back-haul outdoor scenario

In this section, we give some figure of merits (FOM) of a Dband coherent transceiver deployed in an outdoor environment. First the performance of the PHY layer were first assessed to determine the best set of parameters: coding rate, modulation order and modulation shape given the SNR, the PN level and the targeted packet error rate (PER) of 10 -2 . Concerning the PHY layer, we assume a perfectly synchronized single carrier modulation. The input packet size of 1500 bytes is implemented with a coding rate ranging from 0.3 to 0.9. Regarding numerical simulations, we further consider two PN levels: no PN σ 2 φ = 0 rad 2 and strong σ 2 φ = 10 -1 rad 2 . The strong PN levels corresponds to a noise floor spectral density K 0 of -100 dB for a system bandwidth of 1 GHz. Resulting spectral efficiency goes from 0.6 bps/Hz at -0.8 dB SNR, to 7.2 bps/Hz at 29.7 dB SNR without PN and 5.5 bps/Hz at 28.4 dB SNR for strong PN. The system is operated in the Dband, with a BW of possibly several tenths of GHz, divided in 1 GHz channels. The effective BW of the signal is 800 MHz with a 20% overhead due to the control plane. The maximum reachable throughput is respectively 6.2 and 4.4 Gbps/channel when σ 2 φ = 0 and strong PN are considered. These numerical evaluations assume a city center environment while targeting a high capacity mesh back-haul application. The system performance is characterized for an outdoor mesh back-haul scenario, which could be exploited to feed an ultra-broadband mobile access network or fixed wireless access (FWA) network. The sub-THz devices are supposed to be installed on lampposts, located along the streets at 8 meters above the ground, in order to transport the data streams from a fiber PoP (Point of Presence) to dense local high-speed access points (AP) that either provide mobile or FWA typically at a lower frequency. The antenna beam is supposed to be perfectly aligned on the dominant propagation path, which is either the direct LoS path or a reflected path. Other transmission parameters are given in Table I. The propagation paths and losses are predicted from the estimated SNR which is mapped to the spectral efficiency. A channel bonding scheme could increase the aggregated throughput and allow to benefit from the available free spectrum offered in the sub-THz bands. The choice of a channelization of 1 GHz is motivated by current state-of-the-art (SOTA) of analog-to-digital converters

The heat maps in Fig. 1 illustrates how the lamppost-tolamppost link performs depending on the distance, the propagation condition, the PN level and the selected modulation scheme. The illustrated environment is a square in the middle of a dense urban center with medium tree density. The peak throughput per channel is computed between two aligned sub-THz devices; the first one at the center of the figure; the second one at any pixel of the map. Note that the actual lamppost locations, where this second device could precisely be positioned, are shown by black dots. The connectivity is strongly affected by building and foliage obstructions, while LoS propagation and strong clear reflections lead to the best performance. These throughput heat maps demonstrate that optimizing the modulation and demodulation schemes for PN channels results in significant performance gains for future sub-THz systems.

C. Mesh indoor networks

The optical fiber is the best solution to deliver strong capacity in the different blocks of a venue, large ones like e.g. airports, railway stations, stadium, campus, or malls, but a wireless mesh back-haul based on sub-THz technologies may be the final link to some fixed, portable AP under the assumption of efficient antenna auto-alignment and dynamic routing algorithms. As discussed, strong obstructions drasti- For the evaluation of the link capacities we assume a link-layer based on LDPC coded P-QAM with a model of commercial hall. Fig. 2 depicts the two-floor 120m×80m building, which is composed of a large entrance area, large alleys with few obstacles (booths, vegetation, and pillars), and a succession of shops. Each floor is five meters high. A mesh sub-THz network has been designed to feed the wireless AP located at the first floor and entrance area. Two different kinds of AP are considered: those inside the shops, which require a sub-THz relay to be positioned on the outer surface of the shop window; and those connected to a portable/mobile sub-THz relay installed outside the shops. Following radio link are evaluated: i) node-to-node (N2N); ii) node to the fixed shop relays at height 4 meters (N2R). The sub-THz system parameters are given in Table I. For this scenario we assume a strong PN. The propagation model is enabling a maximum of 2 reflections and 1 diffraction along each ray path. Reflections on the ceiling or floor surfaces are not allowed as they are likely to be obstructed. Shop windows are assumed as opaque surfaces for considered frequency.

The peak throughput on each radio link is predicted with the antennas aligned on either the direct-path or the strongest propagation path. Fig. 2 depicts the calculated throughput for all mesh connections and the best shop-relay connections. Every mesh node is maximum 2-hop distant from the PoP having a greater than 4 Gbps/channel peak throughput. All shop relays are attached to the mesh network with more than 2 Gbps/channel peak throughput. This study has been detailed in [START_REF] Gougeon | Assessment of sub-THz Mesh Backhaul Capabilities from Realistic Modelling at the PHY Layer[END_REF] and has demonstrated the capability of sub-THz technology to address indoor mesh back-haul scenario.

D. Example of D2D

For this last scenario, we assume a short-range transmission with complexity and energy constrained devices. To meet high throughput while considering simple on-off keying (OOK) modulation as well as non-coherent receiver, spatial multi-Fig. 2: Connections between mesh nodes, and to the shoprelays.

plexing is investigated as depicted in Fig. 3. In this case each spatial stream is transmitted with a directive antenna. A common challenge is to design a compact system with a low aperture antenna to limit the interference. A recent work has demonstrated that in case of non-coherent demodulation, a trade-off exists between the level of interference between spatial streams (antennas) and the received power when a digital joint demodulation is performed [START_REF] Bicaïs | Design of MIMO Systems with Energy Detection Receivers for Future Sub-TeraHertz Applications[END_REF].

The main simulation parameters are a carrier frequency of 145 GHz, a bandwidth of 2 GHz, an antenna gain of 32 dBi with a beam width of 3 • and side lobe level at -20 dBi and a transmission range of 5m. Other parameters are assumed identical to the indoor scenario. A uniform linear antennas arrays N × N MIMO system is considered. The specification of the antenna is extracted from [START_REF] Manzillo | High-gain D-band Transmitarrays in Standard PCB Technology for Beyond-5G Communications[END_REF] which describes the design of a high-gain antenna for the D-band with transmitarrays and PCB technology. Further, a polar code scheme is considered to achieve channel coding gain and low packet error rates (PER). The polar code, with a packet size of 512 bytes and a coding rate of 9/10, is decoded with a successive cancellation decoder. The simulated PER with channel coding have been obtained by Monte-Carlo simulations and are reported in Fig. 4 as a function of the level of interference characterized by the following illumination matrix:

    1 1 ρ ... ρ 1 1 1 . . . . . . ρ 1 1 . . . ρ . . . . . . . . . . . . 1 ρ ... ρ 1 1     .
(

) 1
where ρ models the residual interference due to side lobes of the antennas with ρ 2 = -40 dB. We define κ as the number of diagonals whose elements are 1. If κ = 1 then the antennas are almost perfectly spatially multiplexed. Conversely, a large value of κ implies that the many channels are strongly correlated.

- Fig. 4: PER performance for polar coded systems with coding rate 9/10 and OOK modulation scheme Subsequently, it is shown that an 8 × 8 MIMO system may communicate over 5 m in the D-band with a PER below 10 -4 , an antenna transmit power P ATx ≤ -43 dBm and a spatial occupation ≤ 19 cm . The spatial occupation relies on the form factor of the antenna array based on figures from [START_REF] Manzillo | High-gain D-band Transmitarrays in Standard PCB Technology for Beyond-5G Communications[END_REF]. The data-rate of the system is 7.2 Gbit/s for a 2 GHz band. A channel bounding scheme, benefiting from the large available sub-THz bands spectrum, could increase the aggregated throughput. It can be concluded that MIMO systems using energy detection receivers may achieve high spectral efficiency wireless communications in sub-THz bands with low power and low complexity RF architectures. Regarding short-range communications, it is worth mentioning a different numerical application where simulation parameters remain the same but d 0 = 50 cm. It results that a spatial occupation ≤ 1.8 cm and an antenna transmit power P ATx ≤ -62 dBm ensure a PER below to 10 -4 .

E. Summary

To sum up the requirement and constraints of the three aforementioned scenarios, Table II reports the key performance indicators that will be cross correlated in the next section with available CMOS and III-V technologies. The maximum achievable throughput is given for strong PN and no PN conditions. The transmit power is given per channel and per antenna. In this section, their potential and limits are studied for connectivity applications, taking into account connectivity roadmap defined in H2020 NEREID CSA project [START_REF] Nereid H | Deliverable 8.3: Technology Outlook in Nanoelectronics and harmonization of NEREID roadmaps with International Roadmaps[END_REF]. For the sake of comparison, process technology characteristics can be considered w.r.t. building blocks performance, such as PA, low noise amplifier (LNA), voltage-controlled oscillator (VCO), and transceivers as depicted in Fig. 5.

Output power ability depends on Bias Voltage (BV) and I max current. High Speed Digital Integration capability is inversely proportional to the size and efficiency (transit time / current) of transistors. The selectability is the ability to switch RF signals with high isolation and low loss. The linearity of the transistors is given by their output power versus input power relation (x 2 or e x ). Matching property is considered for two minimum size transistors in a differential structure. Substrate resistivity and Thick Metal play a role on High Quality (HQ) passives limiting pulling effect. The ratio F t /NF is a FOM given by the F t and the NF min of the technology (F t for high frequency digital applications and NF min for receiver noise). F max represents the G p = 0 dB point, involving applications maximum frequency should be under F max /3 for performance achievements. The NEREID connectivity roadmap has also considered the three main application domains, namely D2D, indoor and outdoor applications, and, figures are given for the 5-and 10-year terms (medium-to long-term). Based on this analysis, the connectivity requirements for each process technology characteristics have been extracted and are illustrated in Fig. 6, the main parameter being output power.

B. Position of silicon and III-V technologies

Silicon-based processes are compared to III-V / InP based HBT ones and FOMs are depicted in Fig. 7. Si-Based FDSOI, InGaAs/Si, B55 and DOT 7, as well as III-V InP offer interesting properties to address the mmWave bands. The benefits of B55, FDSOI and InGaAS/Si are the ability to allow integration of high speed signal processing. Depending on the required output power, B55 and III-V InP have interesting capabilities (see e.g [START_REF] Hamada | Millimeter-wave InP Device Technologies for Ultrahigh Speed Wireless Communications toward Beyond 5G[END_REF]). RF-SOI and new GaN/Si bring RF power and selectability.

C. Case of PA

For each scenarios given in Table II, potential technologies of interest are considered from power amplification side. As the frequency of interest is high w.r.t to F max , only "A-Class" PA are considered: the operation frequency is less than F max /3, but still no current technology is for 5 × F max which would made "AB-Class" possible. On the output power side, we have to consider that it is transmitted either isotropically or focusedbeam, this latter requiring a multi-patch antenna, where a single antenna is used for relatively isotropic radiation. Thereby, focused-beam are preferred for mmWave communications, as a group of patches, 4 (2 × 2) or 16 (4 × 4) [START_REF] Shahramian | The Limits of Antenna and Interconnect -Integration for Large-Scale Phased Arrays[END_REF] namely a tile, is fed by an individual PA, reducing individual PA output power needs. Phased-array or transmit-array architectures are combining individual power into radiated power, built upon phase-shifter on the signal path or primary radiated signal to a transmit-array. Hence, individual PA output power sum is expected as the final radiated power, and as low as P P A = 0 dBm to 10 dBm are sufficient. For instance, 25 dBm radiated power can be addressed with P P A = 13 dBm 4-tile array or P P A = 7 dBm 16-tile array, this latter being compatible with silicon technologies, as 8 dBm/140GHz is currently viewed as achievable.

FOMs from an Indoor WiGig application are also given for comparison. It should be mentioned that similar applications are envisaged or already deployed in the V-band. Moreover, it has to be noticed that for D-band the propagation loss penalty is only 7.35 dB w.r.t V-band and could be negligible or even favorable to D-band for range above 0.5 km. Moreover, benefits of the channel bounding in the D-band clearly show up, as the BW in the V-band is limited by worldwide regulation (≈ 5 GHz).

IV. CONCLUSION

Competition between technologies, III-V and Si, has been demonstrated as PA output power can be drastically reduced to cover scenarios that have been investigated benefiting from the directivity gain of multiple tiles antenna arrays. Nevertheless, the gain of the PA stage must also be considered as not too many amplification stage are expected to maintain reasonable linearity, cost and consumption. An improvement by a factor 2 for F t , and F max , would definitively help silicon to become a definitive competitor for Indoor and D2D wireless connectivity [19]. The cost will be the main key factor to earn market shares in these market fields. BiCMOS and RF CMOS processes for Outdoor connectivity could also compete as they can also embed some processing thanks to the better integration they offer. Indeed, on this market, the trend is not to do SoC (System-on-Chip), but rather SiP (System-in-Package). Therefore doors are still open for hybrid integration. RF SOI CMOS and FD-SOI have already proven to be adequate up to 40 GHz, providing better integration and overall cost than SiGe or InP HBT, but could also meet the specifications for B5G from mmW to sub-THz connectivity.

Fig. 1 :

 1 Fig. 1: Comparison of standard and proposed solutions when impaired by PN. cally impact propagation, thus imply deployment in a clear space.For the evaluation of the link capacities we assume a link-layer based on LDPC coded P-QAM with a model of commercial hall. Fig.2depicts the two-floor 120m×80m building, which is composed of a large entrance area, large alleys with few obstacles (booths, vegetation, and pillars), and a succession of shops. Each floor is five meters high. A mesh sub-THz network has been designed to feed the wireless AP located at the first floor and entrance area. Two different kinds of AP are considered: those inside the shops, which require a sub-THz relay to be positioned on the outer surface of the shop window; and those connected to a portable/mobile sub-THz relay installed outside the shops. Following radio link are evaluated: i) node-to-node (N2N); ii) node to the fixed shop relays at height 4 meters (N2R). The sub-THz system parameters are given in TableI. For this scenario we assume a strong PN. The propagation model is enabling a maximum of 2 reflections and 1 diffraction along each ray path. Reflections on the ceiling or floor surfaces are not allowed as they are likely to be obstructed. Shop windows are assumed as opaque surfaces for considered frequency.The peak throughput on each radio link is predicted with the antennas aligned on either the direct-path or the strongest propagation path. Fig.2depicts the calculated throughput for all mesh connections and the best shop-relay connections. Every mesh node is maximum 2-hop distant from the PoP having a greater than 4 Gbps/channel peak throughput. All shop relays are attached to the mesh network with more than 2 Gbps/channel peak throughput. This study has been detailed in[START_REF] Gougeon | Assessment of sub-THz Mesh Backhaul Capabilities from Realistic Modelling at the PHY Layer[END_REF] and has demonstrated the capability of sub-THz technology to address indoor mesh back-haul scenario.

Fig. 3 :

 3 Fig. 3: System model for the D2D scenario.

  Antenna transmit power PA Tx (dBm) PER ∀N, κ = 1 N = 4, κ = 3 N = 6, κ = 7 N = 8, κ = 9

Fig. 5 :

 5 Fig. 5: Process technology targets. III. SILICON BASED TECHNOLOGIES FOR CONNECTIVITY A. Overview and road-maps Nowadays, silicon-based technologies offer low cost compromises for RF and mmWaves applications.In this section, their potential and limits are studied for connectivity applications, taking into account connectivity roadmap defined in H2020 NEREID CSA project[START_REF] Nereid H | Deliverable 8.3: Technology Outlook in Nanoelectronics and harmonization of NEREID roadmaps with International Roadmaps[END_REF]. For the sake of comparison, process technology characteristics can be considered w.r.t. building blocks performance, such as PA, low noise amplifier (LNA), voltage-controlled oscillator (VCO), and transceivers as depicted in Fig.5.Output power ability depends on Bias Voltage (BV) and I max current. High Speed Digital Integration capability is inversely proportional to the size and efficiency (transit time / current) of transistors. The selectability is the ability to switch RF signals with high isolation and low loss. The linearity of the transistors is given by their output power versus input power relation (x 2 or e x ). Matching property is considered for two minimum size transistors in a differential structure. Substrate resistivity and Thick Metal play a role on High Quality (HQ) passives limiting pulling effect. The ratio F t /NF is a FOM given by the F t and the NF min of the technology (F t for high frequency digital applications and NF min for receiver noise). F max represents the G p = 0 dB point, involving applications maximum frequency should be under F max /3 for performance achievements. The NEREID connectivity roadmap has also considered the three main application domains, namely D2D, indoor and outdoor applications, and, figures are given for the 5-and 10-year terms (medium-to long-term). Based on this analysis, the connectivity requirements for each process technology characteristics have been extracted and are illustrated in Fig.6, the main parameter being output power.

Fig. 6 :

 6 Fig. 6: Connectivity roadmap.

Fig. 7 :

 7 Fig. 7: Position of Si-Based and III-V processes.

TABLE I :

 I Simulation parameters.

	Parameters	Notation	Outdoor	Indoor N2N Indoor N2R
	Carrier frequency	fc		150 GHz
	Channel bandwidth	B		1 GHz
	Allocated bandwidth	Ba		0.8 GHz
	Thermal noise	N0		-174 dBm/Hz
	Noise figure	N f		10 dB
	Transmit power	P Tx	30 dBm	19 dBm	19 dBm
	Tx/Rx antenna gain	(g Tx , g Rx ) (25, 25) dBi	(25, 25) dBi	(25, 8) dBi
	Implementation loss	L		2 dB
	Control plane overhead	µ		20 p.c

TABLE II :

 II Summary of scenario key performance indicators.

	Scenario	Throughput no PN -strong PN	Channel	Transmitted Power	Modulation/Coding	Range
	Outdoor	4.4 -6.2 Gbps/ch	1 GHz	25 dBm	Coherent P-QAM LDPC	∼300m
	Indoor node-to-node	4.4 -6.2 Gbps/ch	1 GHz	19 dBm	Coherent P-QAM LDPC	∼50m
	Indoor node-to-relay	2 -2.8 Gbps/ch	1 GHz	19 dBm	Coherent P-QAM LDPC	∼15m
	D2D MIMO 8 × 8	7.2 Gbps/ch	2 GHz	-43 dBm per antenna	Non-coherent OOK Polar code	<5m