Keywords: distributed learning, information theory, classification, Information Bottleneck, logarithmic loss, resource allocation

Consider a setup in which multiple sensors or nodes work together to track, classify, or otherwise infer a quantity or label of interest. This is a configuration commonly found in many applications, e.g., environmental monitoring, self-driving cars, medical diagnosis. In practical applications, developing analytical models for such problems is challenging due to the complex relationships between the data observed and the target variable that needs to be inferred. Machine learning based models, such as neural networks (NN), have proven to be adept at modelling such complex relationships from data, through a so-called training phase. However, these sensors are usually restricted in terms of the data they are allowed to transmit. This restriction could be due to constraints on the communication network, found in environmental monitoring, or privacy constraints, found in medical problems. To overcome these issues, models that can be trained and used in a distributed manner need to be developed.

In this thesis, we provide a framework, named in-network learning, that can be used in such a distributed setting during both the training and the inference phase. The framework is made up of an architecture that describes how the multiple NNs should interact with each other during training and inference, as well as a loss function that can be used to jointly optimise the parameters of the NNs in a distributed manner. Inspired by the information bottleneck method, the loss function looks to train the NN to achieve the best trade-off between the accuracy of the predictions, under logarithmic loss, and the complexity of the information passed between the nodes of the communication network, measured by their minimum description length (MDL). The loss function and architecture take into account the network topology and can be applied to any communication network that can be modelled by a directed acyclyc graph.

Finally, we consider the problem of scheduling and resource allocation in such distributed inference networks. Inference models are commonly used in constrained communication networks. In this setting, the resources of the communication network, such as power or bandwidth, can be redistributed to improve network performance.

Due to the lack of prior knowledge, deciding how to properly allocate available resources, based on only distributed available data, can be a challenging task. The issue appears due to the fact that no one node can provide a definitive picture of the system.

iii Each data observing node can assess the usefulness of its own data; however, it does not have knowledge of the other nodes data. The decision node has an understanding of the usefulness of each node relative to the other nodes, but only based on compressed representations. As such, in this thesis, we provide a scheduling and power allocation algorithm that overcomes this issue by making use of both local assessments, in which each node assesses the quality of its own data, and global assessments, in which the decision node assesses the usefulness of each node based on compresses representations.

Our scheduling and power allocation algorithms then combine these two assessments in a suitable manner to improve the performance of the network.

List of Figures

2.6

The trade-off between the relevance and complexity in the IB problem.

The solution to the IB problem of (2.26) results in a relevance-complexity pair on the bound of the feasible region (grey area). Figure taken from [START_REF] Zaidi | On the information bottleneck problems: Models, connections, applications and information theoretic views[END_REF].

2.7

The diagram shows the variation information bottleneck when we parameterized the distributions using NNs. Given a sample x i the encoding NN outputs the parameters of a distribution, in this case a Gaussian.

Samples are then drawn from this distribution and an estimate of y i is obtained by the NN decoding for each sample. . . . . . . . . . . . . . . . Since the sensors are tracking the same vehicle, the data collected by the sensors are correlated; however, the exact relationship between the data we observe and the data we want to predict is difficult to know in advance. As such, we assume that initially the sensors collect data from which we can train a model to learn the relationship between the observed data and the information we want to predict, that is, tracking the vehicle.

We refer to this period of time as the training phase. The most natural approach to learning such a model would be to collect the data samples from the sensors to a central location, where a model can be learnt based on all the data. Unfortunately, sensors can be deployed in an area where the communication infrastructure is limited and intervention is not possible after deployment to improve the infrastructure or collect the data [START_REF] Majid | Applications of wireless sensor networks and internet of things frameworks in the industry revolution 4.0: A systematic literature review[END_REF]. As such, transferring large amounts of data to a central location might not be possible due to communication constraints. Another issue with centralising data is that sensitive information can be contained in the collected data and its transfer could violate privacy restrictions [START_REF] Horvitz | Data, privacy, and the greater good[END_REF]. Due to these reasons, both the training phase and the deployment of the model, also termed the inference phase, need to take place in a distributed manner, i.e., without the exchange of raw data. This problem is sometimes referred to as the distributed training and inference problem.

More generally, in this thesis, we assume that there are some devices, which we denote as edge nodes, that observe data related to a variable of interest that needs to be inferred at a central decision node or fusion centre. The edge nodes are connected to the central node either directly or through some intermediary nodes. We assume that the topology of the network can be modelled by an acyclic directed graph. The distribution of the data is not known and only a distributed data set is available; that is, different parts of the data are available at different nodes. We equip each node with a neural network (NN). This is done due to the success of machine learning (ML) techniques in learning, from samples, complex relationships between observed data and a target variable in areas such as computer vision [START_REF] Zou | Object detection in 20 years: A survey[END_REF], image processing [START_REF] Pluim | Mutual-information-based registration of medical images: a survey[END_REF], robotics [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF], and natural language processing [START_REF] Vinyals | A neural conversational model[END_REF].

The question is, given some measures of the performance of the model, what is the best way to train such a model? Clearly, removing any of the relevant data can only lead to a reduced prediction performance. However, on the other hand, extracting and transmitting data that are not relevant to the task at hand can put a strain on the communication network or even degrade the performance of the network. Thus, intuitively, nodes need to extract the smallest number of features that collectively give the best prediction performance. Deciding which features to extract needs to be done without explicit coordination between the nodes. How should that be performed optimally without sharing raw data? In particular, how should each node process

information and what should it transmit to other nodes?

Contributions

In this thesis, we focus on the distributed training and inference problem. We discuss how multiple nodes, that observe data related to the same event, can work together to jointly train a model that can be used in a distributed manner during the inference phase through a novel framework named in-network learning (INL).

INL for a star topology, in which the nodes that observe data are directly connected to the decision node, was first introduced and analysed in [START_REF] Aguerri | Distributed variational representation learning[END_REF][START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF]. We build on these results by providing a novel experimental comparison of INL with other distributed learning algorithms, namely those of [START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF][START_REF] Gupta | Secure training of multi-party deep neural network[END_REF]. These results were published in [START_REF] Moldoveanu | On in-network learning. a comparative study with federated and split learning[END_REF].

The extension of the INL to network topologies that can be modelled with a directed, acyclic graph is proposed and studied in the patent document [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by A. Zaidi, K.

Scaman and P. Escamilla. In this thesis, we derive a suitable loss function for such a model. The approach to the derivation of the loss function is borrowed from [START_REF] Aguerri | Distributed variational representation learning[END_REF],

i.e., first we assume that joint distributions are known and proceed with modelling the problem as a network source coding one under logarithmic loss measure and finding suitable (achievable) rate-distortion trade-offs, then we derive a variational inner bound, and finally we parametrize that lower/inner bound using neural networks.

This work was published as part of [START_REF] Moldoveanu | In-network learning for distributed training and inference in networks[END_REF].

Finally, we consider the practical problem of deploying INL for a star topology in a constrained wireless setting; an example would be the deployment in wireless sensor networks (WSNs) [START_REF] Zhengzhong | Wireless sensor networks for living environment monitoring[END_REF]. The performance of WSNs is significantly restricted by its available radio resources and the way in which the resources are allocated between nodes. Allocating available resources, when we only have access to a dataset, is impeded by the distributed nature of the data and the unknown relationships between the data observed by each sensor. The patent [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF] already suggested the idea of having:

(i) each device measuring its own local assessment of how relevant is the data it holds for the inference task at hand and transmitting it to the decision node, (ii) that device also simultaneously sends the output activation values to the decision node, (iii) the decision node combining all received information (activation values from the various devices as well as their individual local assessments) and then forming its own (global) assessment of how relevant each device's observation is, (iv) application of the idea to device scheduling and power allocation (two algorithms for scheduling were already given in [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF]). All these steps were later reported on and detailed in the intermediate version [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]. We differentiate ourselves from these works by describing how the the local nodes can compute their local assessments, as well as providing an appropriate way for the decision node to combine the received local assessments with the activation values to obtain its own global assessment. We also propose an improved scheduling algorithm, as well as a power allocation algorithm, which was not presented in either [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF][START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]. The algorithms are validated experimentally.

Outline

The thesis is organised as follows.

Chapter 2

In Chapter 2 we review some of the results of point-to-point communication, learning, and the connection between the two. We start by reviewing some information-theoretic measures related to both communication and learning, which are used frequently throughout the thesis, including mutual information and entropy. A more in-depth discussion of these measures can be found in [START_REF] Cover | Elements of Information Theory[END_REF]. We then discuss the point-to-point communication problem [START_REF] Shannon | A mathematical theory of communication[END_REF], highlighting how mutual information, introduced previously, can be used to measure the number of bits (rate) required to transmit information between two devices. Afterward, we discuss the learning problem, specifically from the perspective of the information bottleneck (IB) method [START_REF] Tishby | The information bottleneck method[END_REF]. Finally, we review some of the results of [START_REF] Zaidi | On the information bottleneck problems: Models, connections, applications and information theoretic views[END_REF] that connect the IB problem and the point-to-point communication problem.

Chapter 3

In Chapter 3 we study the distributed learning and inference problem in which multiple edge nodes that observe data are connected directly to a decision node that needs to predict some target variable based on the information it receives from the edge nodes. This is a particular case of the learning problem discussed at the beginning of this chapter. We discuss the algorithm proposed by [START_REF] Aguerri | Distributed variational representation learning[END_REF] which seeks to solve this problem, including an in-depth analysis of the training and inference procedure provided in [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF].

We then provide a novel experimental comparison between the algorithm of [START_REF] Aguerri | Distributed variational representation learning[END_REF] and other distributed training algorithms, namely the famous federated learning algorithm of [START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF] and the split learning of [START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF].

Chapter 4

In Chapter 4 we study a generalisation of the problem studied in Chapter 3. More specifically, we consider the case in which the edge nodes that observe data do not communicate directly with the decision node, but through some intermediary nodes.

We analyse the case in which the topology of the network can be modelled by a directed acyclyc graph. The extension of the INL to network topologies that can be modelled with a directed, acyclic graph is proposed and studied in the patent document [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by A. Zaidi, K. Scaman and P. Escamilla. In this chapter, we first derive an attainable trade-off between the complexity of the features extracted by each edge node and the prediction performance at the decision node, which depends on the network topology.

We then discuss how, given a network topology, one can derive a loss function for training the model in a distributed setting using the derived trade-off. We then exemplify the steps by deriving the loss function for an example network topology. The approach to the derivation of the loss function follows similarly to the ones of [START_REF] Aguerri | Distributed variational representation learning[END_REF], i.e., using the obtained (achievable) rate-distortion trade-offs we derive a variational inner bound, and finally we parameterize that lower/inner bound using neural networks. This work was published as part of [START_REF] Moldoveanu | In-network learning for distributed training and inference in networks[END_REF].

Chapter 5

In Chapter 5 we consider the implementation of the algorithm discussed in Chapter 3, in a resource-constrained wireless sensor network. We study the problem of scheduling and resource allocation in the case where the distribution of the data is not known. In [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF][START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF] it was observed that in such a situation it is difficult to schedule and allocate resources, since no one node has the complete picture of the setting. While the decision node receives the data from all the nodes and thus can assess their relative usefulness, the information received by the central node is only a compressed representation of the full data. The edge nodes have access to the full data; however, they cannot assess what other nodes observe. In [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF], the author patented the idea of having: (i) each device measuring its own local assessment of how relevant is the data it holds for the inference task at hand and transmitting it to the decision node, (ii) that device also simultaneously sends the output activation values to the decision node, (iii) the decision node combining all received information (activation values from the various devices as well as their individual local assessments) and then forming its own (global) assessment of how relevant each device's observation is, (iv) application of the idea to device scheduling and power allocation (two algorithms for scheduling were already given in [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF]). In this chapter, we present in detail the ideea of the patent [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF],

which was also described in [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]. We then build on this ideea by describing how the the local nodes can compute their local assessments, as well as providing an appropriate way for the decision node to combine the received local assessments with the activation values to obtain its own global assessment. We also propose an improved scheduling algorithm, as well as a power allocation algorithm. Finally, we present an experimental comparison to showcase the advantages of the presented scheduling and power allocation algorithms. Part of these results are presented in [START_REF] Zaidi | In-network learning: Distributed training and inference in networks[END_REF].

Chapter 6

In Chapter 6 we discuss potential areas of future investigation.

Notation

The following notation will be used throughout the paper. Upper case letters denote random variables, e.g. X; lower case letters denote realisations of random variables, e.g x, and calligraphic letters denote sets, e.g., X. The cardinality of a set is denoted by |X|. For a set of natural numbers S ⊆ K , the complementary set of S is denoted by S c , that is, S c = {k ∈ N : k ∈ K \ S}. For a random variable X, that takes values in X, with probability distribution P X , the shorthand p(x) = P X (x), x ∈ X is used. Boldface letters denote matrices or vectors, e.g., X or x. A sequence of random variables (X k , X k+1 , ..., X j ) is abbreviated by X j k . The sequence X n 1 is sometimes denoted simply as X n , i.e., X n ≜ (X 1 , X 2 , ..., X n ). For random variables (X 1 , X 2 , ...) and a set of integers K ⊆ N, the notation X K designates the vector of random variables with indices in the set K , that is,

X K ≜ {X k : k ∈ K }. If K = ∅ then X K = ∅.
Furthermore, for random vectors with zero mean x and y, the quantities x , x,y and x|y denote, respectively, the covariance matrix of the vector x, the covariance matrix of vector (x, y) and the conditional covariance of x given y. We will use the notation diag([x 1 , . . . , x n ]) to denote an n×n matrix whose diagonal elements are given by [x 1 , . . . , x n ], while the off-diagonal elements are zero. We say that the random variables U, X and Y form a Markov Chain in that order, which we denote as U -X -Y, if p(u, x, y) = p(x, u)p(y|x). Finally, for two probability measures P X and Q X over the same alphabet X, the relative entropy or Kullback-Leibler divergence is denoted as

D KL (P X ||Q X ). That is, if P X is absolutely continuous with respect to Q X , then D KL (P X ||Q X ) = E P X [log(P X (X)/Q X (X))], otherwise D KL (P X ||Q X ) = ∞.

Chapter 2

Connection between Communication and Learning

In this chapter, we discuss a simplified version of the problem discussed in the introduction. We consider that there are only two nodes, one node that observes data, and one node that wants to infer the target variable. This setting greatly simplifies the problem, as now the node observing the data does not need to take into account what other nodes might observe. This chapter is a review of relevant results from the literature.

The aim of this chapter is to give some intuitions that will be used to derive results for the more general case discussed in Chapter 1. For the case where the distribution of the data is known, and the target variable is the observed data, the problem reduces to that of point-to-point communication. This is a famous problem that was tackled by Shannon [START_REF] Shannon | A mathematical theory of communication[END_REF], and in this chapter we formally introduce the problem of point-to-point communication and restate some of the results presented in [START_REF] Shannon | A mathematical theory of communication[END_REF]. We then discuss the problem of point-to-point communication when the target variable is not the observed data. Afterwards we consider the same setup, however, we assume that the distribution of the data is not known and only a dataset is available. We discuss this learning problem from the perspective of the famous information bottleneck (IB) method [START_REF] Tishby | The information bottleneck method[END_REF].

We first discuss the intuition behind the IB problem and its solution for the case in which the distribution of the data is known and discrete. We then present the results of [START_REF] Alemi | Deep variational information bottleneck[END_REF] in which the authors extended the IB for the case in which the distribution of the data does not need to be known and only a set of training samples need to be available. Next, we discuss the connection between IB and the point-to-point communication problem.

We revisit the results of [START_REF] Zaidi | On the information bottleneck problems: Models, connections, applications and information theoretic views[END_REF] that discuss how IB is essentially a compression problem, and then discuss how the rate of compression from point-to-point communication is linked to a key learning metric, namely the generalisation gap. At the beginning of this chapter, we briefly review some information-theoretic measures, which are fundamental to many results in this thesis, before discussing the aforementioned results.

The introduction is very brief, with many details left out; for a comprehensive review see [START_REF] Cover | Elements of Information Theory[END_REF].

Preliminaries

Consider the case in which a person draws a word from a hat and we need to predict what word they drew, with as little additional information from the person who drew the word. We know in advance the finite set of words X = {x 1 , . . . x |X| } from which a person can draw a word. In the case in which we cannot receive any information from the person, if all the words are equally like to be used, then it would be hard to actually predict correctly what the word they drew is. However, if the hat contains only one word, then we can easily predict correctly what the chosen word is, without any additional information. Intuitively, the prediction is affected by the amount of uncertainty that we have about the word that will be drawn, and as a consequence the more uncertain we are, the more additional information we need to receive. Let the probability that any word is chosen be modelled by the random variable X with the probability mass function P X . We would like to be able to quantify the amount of uncertainty contained in X, and we will define this measure of uncertainty as H(X). In his seminal work [START_REF] Shannon | A mathematical theory of communication[END_REF], Shannon suggested the following three conditions that such a measure of uncertainty should satisfy. (2.1)

The Boltzamn-Shannon entropy (which we will refer to as entropy) was shown to be the only function 1 that satisfies the three conditions mentioned above.

Definition 2.1.1. Let X be a discrete random variable distributed according to P X . The entropy of X is defined as

H(X) = - x∈X p(x) log p(x). (2.
2)

The base of the logarithm sets the scale of the measure. Unless otherwise stated, throughout this thesis, we will assume that the logarithm is in base 2 and we will measure the entropy in bits. Intuitively, the entropy of base 2 can be seen as the minimum number of yes or no questions that we need to ask to figure out what the drawn word is, on average. The definition of entropy can easily be extended to the case of joint entropy, which is defined on a set of random variables X 1 , . . . , X N and measures the joint uncertainty.

Definition 2.1.2. For a set of N discrete random variables distributed according to P X 1 ,...,X N , the joint entropy of the set is defined as

H(X 1 , . . . , X N ) = - x 1 ,...,x N p(x 1 , . . . , x N ) log p(x 1 , . . . , x N ). (2.3)
Closely related to entropy is the concept of conditional entropy. Consider again the case in which a person draws a word from the hat that we need to guess; however, we now obtain some additional information from the person, for example, the number of letters in the word, which we will denote by Y. If all words in the space X have the same number of letters, then this additional information does not reduce the uncertainty; that is, Y does not provide information on X. If all the words in the space X have, each, a different number of letters, then given Y we can know X. The conditional entropy H(X|Y) of a random variable X given another Y, measures the uncertainty of X that remains once we determine the value of Y. From the above definition, it can be obtained that

I(X; Y) = I(Y; X) = H(X) + H(Y) -H(X, Y), (2.6)
I(X; Y) = H(X) -H(X|Y) = H(Y) -H(Y|X). (2.7)
A diagram showing the relationship between mutual information, entropy, and conditional entropy can be seen in Figure 2.1.

Finally, we introduce the concept of Kullback-Leibler(KL) divergence. This measure, also known as relative entropy, quantifies the distance between two distributions P X and Q X defined over the same space.

Definition 2.1.5. The KL divergence between two discrete probability distributions P X and Q X is defined as

D KL (P X ||Q X ) = x p(x) log p(x) q(x) . (2.8)
Additionally, KL divergence is related to mutual information through the following well-known equality

I(X; Y) = D KL (P(X, Y)||P(X)P(Y)). (2.9)
An interesting property of the KL divergence is that it is a positive value and equals zero if and only if P X (X) = Q X (X). We denote random variables (Y, X, U) for which p(x, y, u) = p(x, y)p(u|x) as Y -X -U, and we say that they form a Markov chain.

So far, we have only talked about discrete random variables; however, the KL divergence can also be defined for continuous random variables.

Definition 2.1.6. The KL divergence between two continuous probability distributions P X and Q X is defined as

D KL (P X ||Q X ) = X p(x) log p(x) q(x) dx.
(2.12)

Similarly, using (2.9), the concept of MI can be extended to continuous random variables.

Definition 2.1.7. Let (X, Y) ∼ P X,Y be two continuous variables with marginal distributions p(x) = Y p(x, y) and p(y) = X p(x, y). Mutual information between X and Y is defined as

I(X; Y) = X Y
p(x, y) log p(x, y) p(x)p(y) dxdy.

(2.13)

We have introduced the fundamental measures required throughout this paper.

Point-to-Point Communication

In point-to-point communication, we want to send information from one node, the sender, to another node, the receiver, over some imperfect medium of communication, the channel. Shannon [START_REF] Shannon | A mathematical theory of communication[END_REF], formulated this problem mathematically using the communication model shown in Figure 2.2. The model consists of a discreet source of messages X, a sender, a receiver, and a discreet communication channel that connects the sender to the receiver. The source X has a probability mass function P X , which takes values in the alphabet X. A channel is defined by its input alphabet W, the output alphabet Ŵ, and the transition probability p( ŵ|w). We will consider the case in which the channel is memoryless, that is, the probability of the output of the channel depends only on the input at the time. We consider that the sender observes a sequence of n independent messages, X n := {X i } n i=1 ∼ n i=1 P X , which must be communicated to the receiver. The sender observes these messages and then encodes them so that they can be transmitted over the channel. For simplicity, in Figure 2.2 and the rest of this thesis, we assume that the sender can use the channel n times to transmit information; this is not always the case, and we can find more general results for the case where the number of channel uses is not equal to the number of messages observed in [START_REF] Cover | Elements of Information Theory[END_REF]. As such, the sender needs to find an encoding function that, given X n , outputs a representation W n that can be transmitted over the channel. The representation W n is distorted by the channel transition probability p( ŵ|w) and the receiver obtains Ŵn . The receiver then, using a decoding function, obtains an estimate of X n , denoted by Xn , based on the received Ŵn .

Given a fixed source P X and a channel p( ŵ|w), the aim is to find the encoder and decoder functions for which X can be reconstructed by the receiver, with the smallest error. It A key result of Shannon [START_REF] Shannon | A mathematical theory of communication[END_REF] was to show that this separation and disjoint design between source coding and channel coding is optimal for the case where messages are encoded and decoded in large sequences, that is, n is large. A formal statement of the theorem will be discussed in Section 2.2.2. We will discuss the problem of source coding and channel coding, relating the amount of bits that need to be transmitted, with the accuracy of the reconstruction of the original messages at the receiver. In future sections, we relate the problem of source coding with that of learning and discuss how choices in the learning problem can have an impact on the communication constraints that the inference scheme would place on the communication network.

Source Coding

Source coding, or source compression, aims to obtain a compressed representation of the original messages X n , such that the original messages can be recovered from the compressed representation, within some distortion measured by a chosen distortion function d(x, x).

Loosely speaking, rate-distortion (RD) theory aims to characterise the minimum number of bits required to be transmitted from the sender to the receiver such that the receiver can recover the original messages within some distortion level. More specifically, consider the following encoding function Let the rate distortion function R(D) be defined as the infimum of the rates R such that (R, D) is achievable. Shannon [START_REF] Shannon | A mathematical theory of communication[END_REF] has shown that R(D) can be computed by solving the following constrained optimisation problem

f : X n -→ U n with U n = {1, . . . ,
R(D) := min f,h I(X, X) (2.17) s.t. D f,h (X) ≤ D.
Although RD theory does not explicitly describe how to find ( f, h) that solves the above optimisation problem, this bound can be used to inform the design of the pair ( f, h).

Channel Coding

Once the source is compressed, the obtained representation must be transmitted through the channel. The amount of information that can be transmitted over a channel during one use is limited and is usually quantified by the channel capacity. Specifically, the capacity of a channel is defined as the maximum rate in bits per channel use at which information can be transmitted with an arbitrarily low probability of error. The capacity of a channel was shown to be equal to the maximum mutual information between its input and output, which is denoted as the information channel capacity. with a rate-distortion function R(D) and a channel p( ŵ|w) with capacity C, the following statements hold

• if R(D) < C, then (r, D) is achievable for r ≤ R(D), • if (r, D) is achievable, then r ≤ C.
Throughout this thesis, we will invoke the principle of capacity-achieving codes.

This means that we will assume that we can find channel codes that will allow us to transmit information at rates equal to the channel capacity.

Remote Source Coding

Now, we extend the source coding problem to the case in which the data the receiver wants to recover are not the observed data X n . Instead, the receiver is interested in recovering another variable, Y n , which is correlated with the observed data. More specifically, the sender wants to compress the sequence of messages X n , so that the receiver can recover another variable Y n , which is related to X n through some distribution P X|Y . We are still interested in characterising the R(D) function; however, now the distortion is measured between Y n and the estimate Ŷn .

Let the decoder at the receiver be defined as

g : U n -→ Ŷn . (2.19)
The diagram of the communication model is shown in Figure 2.4. In [START_REF] Witsenhausen | Indirect rate distortion problems[END_REF][START_REF] Dobrushin | Information transmission with additional noise[END_REF] it was shown that the R(D) function in this case is given by

R(D) = min f,g I(X, Ŷ) (2.20) s.t.D f,g (Y) ≤ D.
The proof follows from the fact that the remote source coding problem can be reduced to a direct source coding problem for observable data X n and a different distortion measure. In RD theory, the design of the distortion measure d( x, x) is subject to the choice of the designer. In the next section, we analyse the RD region under a particular distortion function.

Remote Source Coding under Logarithmic Loss

In this thesis, we will focus on logarithmic loss (which was introduced in the context of RD theory in [START_REF] Courtade | Multiterminal source coding with an entropybased distortion measure[END_REF]). Logarithmic loss, or log-loss for short, is a natural distortion measure in the settings in which the reconstructions do not have to be deterministic.

That is, the decoder provides an assessment of the probability of each possible estimate, as opposed to choosing one estimate. Let the random variable Y denote the source with finite alphabet Y to be recovered by the receiver. Furthermore, let P(Y) denote the set of probability measures on Y. The log-loss between y ∈ Y and its reconstruction ŷ ∈ P(Y) is given by

d log (y, ŷ) = log 1 ŷ(y) . (2.21)
Outside RD theory, the measure is widely used in various contexts, including clustering [START_REF] Gur | Variational information bottleneck for unsupervised clustering: Deep gaussian mixture embedding[END_REF] and classification [START_REF] Alemi | Deep variational information bottleneck[END_REF], pattern recognition [START_REF] Shkel | Universal lossy compression under logarithmic loss[END_REF], learning and prediction [START_REF] Cesa-Bianchi | Prediction, learning, and games[END_REF],

image processing [START_REF] Andre | Entropy-based distortion measure for image coding[END_REF], privacy [START_REF] Tripathy | Privacy-preserving adversarial networks[END_REF], and others. In [START_REF] Jiao | Justification of logarithmic loss via the benefit of side information[END_REF], the authors justify the use of logarithmic loss by showing that it is the only loss function that satisfies a natural data processing property that appears in the presence of side information. In another work it was shown that minimising the logarithmic loss for the binary classification problem minimises the risk associated with any other smooth, proper, and convex functions, see [START_REF] Painsky | On the universality of the logistic loss function[END_REF]. Finally, in [START_REF] No | Universality of logarithmic loss in lossy compression[END_REF], the authors show the universality of log-loss over a finite alphabet in fixed-length lossy compression. Essentially, the authors show that, given any discrete source, fixed-length, lossy-compression problem, with some arbitrary distortion measure, there exists an equivalent fixed-length lossy-compression problem with the same alphabet source where the distortion is measured by the log-loss. By equivalence the authors implies that the optimal scheme for one problem is optimal for the other problem, additionally, a good scheme for one problem is also a good scheme for the other problem. These findings, along with the wide range of applications of log-loss, justify its use in our problem setting.

Let X denote the observed data, Y the target variable, and P X,Y their joint distribution.

Let us denote the encoder-decoder model by ϕ(x) = Q(•|x) ∈ P(Y) for any x ∈ X. The expected log-loss of such a model is lower bounded by the conditional entropy of Y

given

X E P X,Y [d log (Y, Ŷ)] = x∈X y∈Y p(x, y) log 1 q(y|x) = x∈X y∈Y p(x, y) log 1 p(x, y) + x∈X y∈Y p(x, y) log p(x, y) q(y|x) = H(Y|X) + D KL (P Y|X ||Q Y|X ) ≥ H(Y|X) (2.22) with equality if an only if Q Y|X (•|x) = P Y|X (•|x).
Now, let the possible stochastic encoding mapping f :

X n -→ U n with |U n | ≤ 2 nR .
Additionally, the possibly stochastic decoder mapping is g : U n -→ P(X n ). Using (2.22) we can lower bound the expected log loss give by (2.16) as

D f,g ≥ H(Y|U). (2.23)
From the above inequality it can be shown that the rate-distortion region of the remote source coding problem under logarithmic loss is given by the union of all pairs (R, D) that satisfy

R ≥ I(U; X) (2.24a) D ≥ H(Y|U) (2.24b)
where the union is over all the auxiliary random variables U that satisfy the Markov chain Y -X -U and, without loss of generality, for which |U| ≤ |X| + 1, see [START_REF] Zaidi | On the information bottleneck problems: Models, connections, applications and information theoretic views[END_REF]. As noted in [START_REF] Harremoes | The information bottleneck revisited or how to choose a good distortion measure[END_REF] the region presented in (2.24) is related to the IB problem; this connection will be discussed in more depth in Section 2.4.

Remote Inference

In this chapter so far we have discussed the case in which the edge node observes samples from the source X and needs to transmit some information to the central node such that it can infer the target variable of interest Y, given that the joint distribution P X,Y is known. We now consider the case in which P X,Y is not known and the encoding and decoding functions need to be learned from a dataset. This problem is sometimes referred to as the remote inference problem, and is shown in Figure 2.5.

For the remote inference problem described, the objective is to find a good mapping from an observed signal X to a target variable Y, based on training samples {(x i , y i )} n i=1 . Although how to define a good mapping is subject to debate [START_REF] Li | Minimax learning for remote prediction[END_REF], the most common approach is to find the mapping that minimises the empirical estimate of some chosen risk. The mapping is made up of two parts, as shown in Figure 2.5. First, we have an In [START_REF] Achille | Information dropout: Learning optimal representations through noisy computation[END_REF], the authors discussed the criteria that a good latent representation U of X should meet, for the problem of inferring Y, which are listed bellow

• U is a function of X, that is, the Markov chain Y -X -U holds,
• Sufficiency: U is sufficient for task Y, i.e. I(U; Y) = I(Y; X),

• Minimality: U discards all variability in X that is not relevant for predicting Y, that is, minimal I(U; X).

Although these criteria give us a method to check if a representation U is a good representation, the problem remains as to how to find such a good representation. For this, we will now introduce the IB method of [START_REF] Tishby | The information bottleneck method[END_REF]. When first introduced, the method was designed for use on discrete variables whose distribution is known. It was in [START_REF] Alemi | Deep variational information bottleneck[END_REF] that the IB problem was adapted for the learning problem. For completeness, we will first present the IB method as first presented in [START_REF] Tishby | The information bottleneck method[END_REF], and then discuss its adaptation to the learning problem.

Information Bottleneck Method

The IB method looks at finding the representation U, satisfying the Markov chain Y-X-U, which is maximally informative of Y (represented by high mutual information I(U; Y)), while minimally informative of X (represented by small mutual information I(U; Y)). The term I(U; Y) is referred to as relevance, and I(U; X) is referred to as complexity. It is important to note that this formulation of the IB problem can be seen as a relaxation of the problem of finding U that meets the criteria listed in Section 2.3. More specifically, the sufficiency condition I(U; Y) = I(Y; X) is relaxed to simply maximising I(U; Y), which due to the Markov chain is always smaller than I(Y; X).

Finding the representation U that maximises I(U; Y) while ensuring that I(U; X) is kept small can be formulated as the following maximisation problem.

∆(R) := max

P U|X :I(U;X)≤R I(U; Y). (2.25)
This problem (2.25) can be solved by reformulating the problem as the following Lagrangian problem L IB s : max

P U|X I(U; Y) -sI(U; X) (2.26)
where L IB s can be called the IB objective and s designates the Lagrange multiplier.

The trade-off between the relevance and complexity of the IB problem can be seen in For a known joint distribution P X,Y and a given value of s, finding the P U|X that maximises L IB s can be done in an iterative manner, similar to the Blahut-Arimoto algorithm of [START_REF] Blahut | Computation of channel capacity and rate-distortion functions[END_REF][START_REF] Arimoto | An algorithm for computing the capacity of arbitrary discrete memoryless channels[END_REF], using the following self-consistent equations

p(u|x) = p(u) exp(-sD KL (P Y|x ||P Y|u ) u p(u)s exp(-sD KL (P Y|x ||P Y|u )) , (2.27a
)

p(u) = x p(u|x)p(x), (2.27b
)

p(y|u) = x p(y|x)p(x|u) = x p(y, x) p(u|x) p(u) . ( 2

.27c)

The IB method starts with some initial condition p 0 (u|x), and iteratively updates the probabilities p(u|x), p(u), and p(y|u) using equations (2.27) until convergence. One disadvantage of this technique is that the IB problem is non-convex, and as such the algorithm is guaranteed to converge only locally. While this analytical solution can only be used when P X,Y is discrete, there are works that provide analytical solutions to the IB problem when (X, Y) are multivariate Gaussian, see [START_REF] Chechik | Information bottleneck for gaussian variables[END_REF][START_REF] Winkelbauer | Rate-information-optimal gaussian channel output compression[END_REF][START_REF] Winkelbauer | The rate-information trade-off for gaussian vector channels[END_REF][START_REF] Meidlinger | On the relation between the gaussian information bottleneck and mse-optimal rate-distortion quantization[END_REF].

Variational Information Bottleneck

In a learning problem we do not know the joint distribution P X,Y , we only have access to a set of training samples {(x i , y i )} n i=1 , which makes computing (2.27) intractable. As a solution, the authors of [START_REF] Alemi | Deep variational information bottleneck[END_REF], present a variational lower bound on the IB objective (2.26), which can then be optimised by parameterizing the distributions using NNs, as we will explain in Section 2.3.2.1.

Let us introduce the variational distribution on U, Q U , and the variational decoder Q Y|U . For simplicity, we denote P := {P U|X } and Q := {Q Y|U , Q U }. We define the variational IB cost function as (2.29)

L VIB s (P, Q) := E P X,Y E P U|X [log Q X|U ] -sD KL (P U|X ||Q U ) . ( 2
In [START_REF] Alemi | Deep variational information bottleneck[END_REF] the authors provide the following relationship between LIB s and L VIB s .

Lemma 2.3.1.

L VIB s (P, Q) ≤ LIB s (P) for all pmfs Q. (2.30) Furthermore, Q that achieves L VIB s (P, Q) = LIB s (P)
is unique and is given by (2.32)

Q * Y|U = P Y|U , Q * U = P U . ( 2

Learning with the Variational Information Bottleneck

We now look at how L VIB s (P, Q) can be used as an objective for a learning algorithm that is capable of training given a dataset {(x i , y i )} n i=1 . Let P θ (U|x) denote the mapping of observation x ∼ X to the latent representation U, which is parameterized by a neural network with parameters (weights and biases) θ. Similarly, we can parameterize the decoding distribution with a NN with parameters ϕ. We denote the distribution induce by the decoding NN by Q ϕ (Y|U). We denote the prior over the latent space as Q φ (U) which does not depend on a neural network. Restricting P, Q to the distributions parameterized by NN, the optimisation of (2.28) becomes max θ,ϕ,φ

E P X,Y E P θ (U|X) [log Q ϕ (Y|U)] -sD KL (P θ (U|X)||Q φ (U)) (2.33)
The parameterised loss function (2.33) can then be approximated, given a dataset (x i , y i ) n i=1 , by its empirical cost given by max θ,ϕ,φ

1 n n i=1 E P θ (U i |x i ) [log q ϕ (y i |U i )] -sD KL (P θ (U i |x i )||Q φ (U i )).
(2.34)

We now illustrate the training procedure for a possible choice of parametric distributions. Consider the case in which the encoder can be chosen as a multivariate Gaussian, that is, P θ (U|x) = N(µ θ , Σ θ ). In such a case, the NN with parameter θ would take as input a sample x i and output the parameters of the Gaussian distribution, that is, the mean µ θ,i and the covariance Σ θ,i . For simplicity, we assume that the covariance matrix is diagonal. Let n u be the dimension of the latent space, that is,

µ θ,i = [µ θ,i,1 , . . . , µ θ,i,n u ] and Σ θ,i = diag({σ 2 θ,i,k } n u k=1
). For the classification problem, we chose the decoder to be a categorical distribution, that is, the neural network with parameters ϕ has a softmax operation in its output layer, which has dimension |Y|. The last layer outputs the probability of each different category of Y. For simplicity, the prior Q φ is chosen as a multivariate Gaussian N(0, I). Calculating the empirical estimate of

E P θ (U i |x i ) [log q ϕ (y i |U i )]
can now be done using the Monte Carlo sampling technique and the reparameterization trick of [START_REF] Kingma | Auto-encoding variational bayes[END_REF] as

E P θ (U i |x i ) [log q ϕ (y i |U i )] ≊ 1 m m j=1 log q ϕ (y i |u i, j ), (2.35
)

with u i,j = µ θ,i + Σ θ,i ϵ j , ϵ j ∼ N(0, I) (2.36)
where m is the number of samples from the Monte Carlo sampling technique and µ θ,i and Σ θ,i are the parameters outputted by the encoding neural network that has observed the sample x i .

The KL divergence term D KL (P θ (U|X)||Q φ (U)) can be calculated analytically, since we have chosen both P θ (u|x) and Q φ as multivariate Gaussian distributions. The term D KL (P θ (U|X)||Q φ (U)) can now be calculated in closed form as follows:

D KL (P θ (U i |x i )||Q φ (U i )) = 1 2 n u k=1 µ θ,i,k -log σ 2 θ,i,k -1 + σ 2 θ,i,k . (2.37)
Together, we have the following cost function for each sample i in the data set

{(x i , y i )} n i=1 L VIB-NN s (ϕ, θ) = 1 m m j=1 log Q ϕ (y i |u i, j ) - s 2 n u k=1 µ θ,i,k -log σ 2 θ,i,k -1 + σ 2 θ,i,k . (2.38)
Finding the parameters that maximise (2.38) can be done using stochastic gradient descent methods (SGD or ADAM [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]). The parameterisation of the distributions using NNs and the sampling procedure described are shown in

Relationship between Communication and Learning

In this section we look at discussing the connection between communication and learning. That is, we discuss how the search for encoding functions that compress the observed data X, by minimising I(X, U), has a double interpretation. On the one hand, I(X, U) is related to the communication rate; that is, by minimising I(U, X), one would need to transmit fewer bits when communicating U. This connection is shown by connecting the IB problem to the remote source coding problem. On the other hand, I(X, U) can act as a regularizer. Intuitively, by minimising I(U, X) one can reduce the difference between the performance of the network on the data it is trained and on new data it has not seen before [START_REF] Zaidi | On the information bottleneck problems: Models, connections, applications and information theoretic views[END_REF][START_REF] Vera | The role of information complexity and randomization in representation learning[END_REF].

Information Bottleneck and Remote Source Coding

The rate-distortion region is characterised by the union of all pairs (R, D) that satisfy (2.24), where U is an auxiliary random variable that obeys the Markov chain Y -X -U.

Using the substitution ∆ := H(Y) -D, the region can be written equivalently as the union of all pairs (R, H(Y) -∆) that satisfy

R ≤ I(U; X) (2.39) ∆ ≥ I(U; Y) (2.40)
The boundary of this region is equivalent to the boundary described by (2.26) if solved for all s. As a consequence, the IB problem can essentially be seen as a remote source coding problem with the distortion measured using log-loss. For a more detailed analysis of the connections between the IB problem and the rate distortion problems, see [START_REF] Zaidi | On the information bottleneck problems: Models, connections, applications and information theoretic views[END_REF][START_REF] Tishby | The information bottleneck method[END_REF][START_REF] Harremoes | The information bottleneck revisited or how to choose a good distortion measure[END_REF][START_REF] Gilad-Bachrach | An information theoretic tradeoff between complexity and accuracy[END_REF]. We can observe that s controls the trade-off between the bit rate constraint R and the desired distortion constraint. As such, by controlling s one can effectively control the overhead that the inference scheme applies on the communication network. This connection has led to multiple lines of work that use the IB method for the problem of inference over a communication channel. For example, in [START_REF] Sun | Adaptive information bottleneck guided joint sourcechannel coding[END_REF] the authors propose an adaptive IB method for designing a joint source and channel encoder-decoder pair. In this method, the parameter s is dynamically adjusted during training to reduce communication requirements while searching for the best balance between communication requirements and network performance. In [START_REF] Shao | Learning task-oriented communication for edge inference: An information bottleneck approach[END_REF] the authors also use the IB method for designing a variable-length feature encoding scheme based on dynamic neural networks that adaptively adjust to different channel conditions.

Connection between Generalisation Gap and Rate

As discussed in Section 2.3 one of the most common ways of choosing the learning model is to find the model that minimises the empirical estimate of the chosen loss. One key performance indicator for such a model is how well the chosen model performs on samples outside the observed dataset, referred to as the generalisation gap. As discussed previously, for an encoder f and a decoder g, under log-loss, the expected loss is given by

C( f, g) := E P X,Y d log (Y, g( f (X))) (2.41)
while, for a given dataset {(x i , y i )} n i=1 , the empirical loss is given by

Ĉ( f, g) := n i=1 d log (y i , g( f (x i ))). (2.42)
The generalisation gap for the model ( f, g) is given by the difference between the expected loss and the empirical loss

gen( f, g) = C( f, g) -Ĉ( f, g). (2.43)
It is commonly believed that one can control the generalisation gap by restricting the complexity of the model. One method is the so-called MDL complexity measure of the model parameters, which is intended to limit the length of the description of the model parameters [START_REF] Hinton | Keeping the neural networks simple by minimizing the description length of the weights[END_REF]. An alternative approach,inspired by RD theory, which is used in this thesis and the IB problem2 , considers minimising the description length of the latent representation U, instead of the model paramters. The two problems have already been

shown to be connected in [START_REF] Achille | Emergence of invariance and disentanglement in deep representations[END_REF], where the authors have shown that, for some models, by decreasing the MDL of the parameters, we automatically improve minimality (see 2.3) and the disentanglement of the latent representation. These two connections, between the MDL of the weights and the generalisation gap, and between the MDL of the weights and the MDL of the latent representations, intuitively show a relationship between the generalisation gap and the MDL of the latent representation. This connection was formalised in [START_REF] Vera | The role of information complexity and randomization in representation learning[END_REF], where the authors have shown that the generalisation gap can be bounded in terms of the complexity of the latent representation, as measured by the empirical estimate of mutual information between X and U obtained from the available data set. Let P(U|X) be the distribution of U given X imposed by the chosen encoder f and let P(n) X be the empirical distribution of X given the data set {(x i , y i )} n i=1 . Furthermore, define P(n) Ŷ|U as the distribution of Ŷ given U imposed by the chosen decoder g, and let P Ŷ|U be the distribution imposed by the optimal decoder g * for the true distribution of the data. The generalisation gap is upper-bounded, as

gen( f, g) ≤ A Î( P(n) X , P U|X ) log n n + B Λ P U|X , P(n) Ŷ|U , P Ŷ|U √ n + O( log n n ) (2.44)
where Î( P(n) X , P U|X ) is the mutual information between X and U under the distribution P(n) X P U|X and Λ P U|X , P(n) Ŷ|U , P Ŷ|U is a function that measures the mismatch between the optimal decoder and the empirical one. A and B are constants. This bound explicitly suggests that the complexity of the latent representations controls the generalisation gap. As such, the IB problem controls the generalisation gap through the variation of the s parameter. This connection between the generalisation gap and the parameter s was experimentally shown in [START_REF] Alemi | Deep variational information bottleneck[END_REF]. In what follows, in this thesis, we will show how the the IB problem can be extended to the distributed case by considering the connection between generalisation and the MDL of the latent representations in an appropriate manner.

Chapter 3

In-network Learning: Star Network

Topology

In this chapter, we build on the results of Aguerri-Zaidi [START_REF] Aguerri | Distributed variational representation learning[END_REF] and Zaidi et al. [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by providing a novel comparison of the algorithm of [START_REF] Aguerri | Distributed variational representation learning[END_REF] with other distributed learning algorithms. More specifically, in this chapter, we study the distributed inference and training problem shown in Figure 3.1. In this setting, multiple nodes observe data related to the same target variable of interest; data that the nodes need to compress and send directly to a central node so that it can predict the target variable. We restrict the problem to the setting in which each node is equipped with an NN to compress the observed data or predict the target variable; the parameters of the NNs need to be learnt from a set of distributed available dataset. This problem was studied in [START_REF] Aguerri | Distributed variational representation learning[END_REF], where the authors proposed an algorithm for training the NN in a distributed manner, the algorithm was named in-network learning (INL) in [START_REF] Moldoveanu | On in-network learning. a comparative study with federated and split learning[END_REF]. In this chapter, we first formally define the problem of Figure 3.1. We also discuss the training procedure for the algorithm of [START_REF] Aguerri | Distributed variational representation learning[END_REF], which was first reported in [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF]. The contribution of this chapter is that we provide a novel comparison of in-network learning with other distributed leaning algorithms, namely the federated learning (FL) algorithm of [START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF] and split learning (SL) of [START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF]. At the end of the chapter, for completeness, we present how [START_REF] Aguerri | Distributed variational representation learning[END_REF] derived the loss function for the INL algorithm presented in this chapter.

Problem Setup

Let us formally introduce the inference problem shown in Figure 3.1. There are nodes J ≥ 1 which possess or can acquire data relevant for inference on a random variable of interest Y, taking values in Y. Let J = {1, . . . , J} denote the set of such nodes, and X j denote the data observed by node j, taking values in X j . The relationship between the random variable of interest Y and the observed ones X 1 , . . . , X J is given by the joint probability mass function P X J ,Y := P X 1 ,...,X J ,Y (x 1 , . . . x J , y), with (x 1 , . . . ,

x j ) ∈ X 1 × • • • × X J
and y ∈ Y. The inference on Y needs to be done at some distant node (say, node (J + 1)) which is connected to the nodes that possess raw data through error-free links of given finite capacities; and has to be performed without any sharing of raw data.

The network may represent, for example, a wired network or a wireless mesh network operated in time or frequency division. The processing at node j ∈ J is a mapping

ϕ j : X j -→ U j ; (3.1)
and that at node (J + 1) is a mapping We study the case in which the mappings given by (3.1) and (3.2) need to be learnt from a set of training data samples {(x 1,i , . . . , x J,i , y i )} n i=1 . The data are distributed so that the samples x j := (x j,1 . . . , x j,n ) are available at node j for j ∈ J and the desired predictions y := (y 1 . . . y n ) are available at the node (J + 1). (3.4)

ψ : U 1 × . . . × U J -→ Ŷ. (3.2) with |U j | ≤ 2 R j .

In-network Learning Solution

Loss Function

A possible suitable loss function was shown to be given by [10]

L NN s (n) := 1 n n i=1 log Q ϕ J (y i |u 1,i , . . . , u J,i ) + s n n i=1 J j=1 log Q ϕ j (y i |u j,i ) -log P θ j (u j,i |x j,i ) Q φ j (u j,i ) , (3.5)
where s is a Lagrange parameter and for j ∈ J the distributions P θ j (u j |x j ), Q ϕ j (y|u j ), Q ϕ J (y|u J ) are variational ones whose parameters are determined by the chosen NNs using the re-parametrization trick of [START_REF] Kingma | Auto-encoding variational bayes[END_REF]; and Q φ j (u j ) are priors known to the encoders.

As described in Section 2.3.2 the encoding distibution can be choosen as multivartiate gaussian denoting by f θ j the NN used at node j ∈ J whose (weight and bias) parameters are given by θ j , for regression problems the conditional distribution P θ j (u j |x j ) can be chosen to be multivariate Gaussian, i.e., P θ j (u j |x j ) = N(u j ; µ θ j , Σ θ j ). For discrete data, concrete variables (i.e., Gumbel-Softmax) can be used instead.

The rationale behind the choice of loss function (3.5) is presented in detail in Section 3.4. However, intuitively, the authors of [START_REF] Aguerri | Distributed variational representation learning[END_REF] have shown that in the regime of large n, if the encoders and decoder are not restricted to use NNs under some conditions1 the optimal stochastic mappings P U j |X j , P U , P Y|U j and P Y|U J are found by marginalising the joint distribution that maximises the following Lagrange cost function [10, Proposition 2]

L DIB s = -H(Y|U J ) -s J j=1 H(Y|U j ) + I(U j ; X j ) . (3.6)
where the maximisation is over all joint distributions of the form P Y J j=1 P X j |Y J j=1 P U j |X j .

Training Algorithm

We now describe the training algorithm for INL which was first reported in the patent [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by Zaidi, Scaman and Escamilla. During the forward pass, every node j ∈ J processes mini-batches of size, say, b j of its training data-set x j . Node j ∈ J then sends a vector whose elements are the activation values of the last layer of (NN j). Due to (3.4) the activation vectors are vertically concatenated at the NN input layer (J + 1). The forward pass continues on the NN (J+1) until the last layer of the latter. The parameters of NN (J+1) are updated using standard backpropgation. Specifically, let L J+1 denote the index of the last layer of NN (J + 1). Also, let, for l ∈ [2 : L J+1 ], w [l] J+1 , b [l] J+1 and a [l] J+1 denote, respectively, the weights, biases and activation values at layer l for the NN (J + 1); and σ is the activation function. Node (J + 1) computes the error vectors

δ [L J+1 ] J+1 = ∇ a [L J+1 ] J+1 L NN s (b) ⊙ σ ′ (w [L J+1 ] J+1 a [L (J+1) -1] f + b [L J+1 ] J+1 ) (3.7a) δ [l] J+1 = [(w [l+1] J+1 ) T δ [l+1] J+1 ] ⊙ σ ′ (w [l] J+1 a [l-1] J+1 + b [l] J+1 ) ∀ l ∈ [2, L J+1 -1] (3.7b) δ [1] J+1 = [(w [2] J+1 ) T δ [2] J+1 ], (3.7c)
and then updates its weight-and bias parameters as

w [l] J+1 → w [l] J+1 -ηδ [l] J+1 (a [l-1] J+1 ) T , (3.8a) b [l] J+1 → b [l] J+1 -ηδ [l] J+1 , (3.8b)
where η designates the learning parameter2 .

Remark 3.2.1.

It is important to note that for the computation of the RHS of (3.7a) node

(J + 1), which knows Q ϕ J (y i |u 1i , . . . , u Ji ) and Q ϕ j (y i |u ji ) for all i ∈ [1 :
n] and all j ∈ J, only the derivative of L NN s (n) w.r.t. the activation vector a

L J+1
J+1 is required. For instance, node (J + 1) does not need to know any of the conditional variationals P θ j (u j |x j ) or the priors Q φ j (u j ).

The backward propagation of the error vector from node (J + 1) to the nodes j, j = 1, . . . , J, is as follows. Node (J + 1) splits horizontally the error vector of its input layer into J sub-vectors with sub-error vector j having size L j , the dimension of the last layer of NN j [recall (3.4) and that the activation vectors are concatenated vertically Remark 3.2.2. Let δ [1] J+1 (j) denote the sub-error vector sent back from node (J + 1) to node j ∈ J. It is easy to see that, for every j ∈ J,

∇ a L j j L NN s (b j ) = δ [1] J+1 (j) -s∇ a L j j        b i=1 log P θ j (u j,i |x j,i ) Q φ j (u j,i )        ;
(3.9)

and this explains why node j ∈ J needs only the part δ [1] J+1 ( j), not the entire error vector at node (J + 1).

Inference Algorithm

We now describe the inference phase for INL which was first reported in the patent [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by Zaidi, Scaman and Escamilla. During this phase, node j observes a new sample x j . It uses its NN to output an encoded value u j which it sends to the decoder. After collecting (u 1 , • • • , u J ) from all input NNs, node (J + 1) uses its NN to output an estimate of Y in the form of soft output Q ϕ J (Y|u 1 , . . . , u J ). The procedure is depicted in Figure 3.2b.

Remark 3.2.3.

A suitable practical implementation in wireless settings can be obtained using Orthogonal Frequency Division Multiplexing (OFDM). That is, the J input nodes are allocated non-overlapping bandwidth segments and the output layers of the corresponding NNs are chosen accordingly. The encoding of the activation values can be done, e.g., using entropy type coding [START_REF] Flamich | Compressing images by encoding their latent representations with relative entropy coding[END_REF].

Comparison to Federated Learning and Split Learning

In this section we compare the proposed INL algorithm with other distributed learning frameworks. We first start by introducing the federated learning (FL) algorithm of [START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF] and the split learning (SL) algorithm of [START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF]. We then discuss the amount of data the three algorithms use for communication during training. Finally we compare the three algorithm experimentally.

Review of Distributed Learning Algorithms

Federated Learning

FL is an alternative approach to the problem of distributed machine learning. The aim of the FL framework is for the nodes to train one model that can then be used, during inference, by each node locally. Each edge device has a set of local data and a copy of the NN model to be trained, common across all devices. Compared to the in-network learning setting presented before, in FL each node has access to the same feature space as the other nodes, however, observes different samples. More specifically, given the same dataset {(x 1,i , . . . , x J,i , y i )} n i=1 , in FL each node j ∈ J observes a subset {(x 1,i , . . . , x J,i , y i )} i∈N j of the samples, with N j ∈ [1 : n]. During the training phase, all edge devices simultaneously train their local copy of the NN on their available dataset.

Then, each edge device sends the learned NN parameters to the central device, called a cloud-or parameter server (PS), which aggregates them, by simply computing their average. The process repeats, every time reinitialising using the obtained aggregated model, until convergence. The rationale is that this way, the model is progressively adjusted to account for all variations in the data, not only those of the local dataset.

Split Learning

SL was presented in [START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF] as an alternative to FL. In SL a two-part NN model is split into an encoder part and a decoder part is learnt sequentially. The central device does not have its own data and hosts the decoder part of the NN. The edge devices have access to a different part of the dataset, split in a manner similar to FL, and a copy of the encoding NN. The central device keeps track of the last trained edge device during training and passes the information to the next device to be trained. In every round, the edge device which will be trained has its parameters initialised using those learnt by the previous device. The parameters are exchanged between the devices directly or through the central node. The edge device then trains the two-part NN, together with the central device, on its distinct dataset using classical gradient descent methods. The edge device passes the data through its NN and sends forward to the central device the activation values of the last layer. The central device uses the received data as input into its NN and outputs an estimate. The central device then backpropagates the error through its NN and sends the error vector back from its input layer to the edge device, which continues the procedure on its own NN. During the inference phase, the edge device observing data connects to the central node and the two jointly perform the inference. SL reduces some of the computation load placed on each edge node by the FL algorithm, while also reducing some of the communication overhead. This was shown in [START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF]. One disadvantage, however, is that the model on each node cannot train simultaneously and as such the model would take more time to train.

Bandwidth Requirements

In this section, we study the requirements in bandwidth of our in-network learning, which was first reported in the patent [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by Zaidi, Scaman and Escamilla. Let q denote the size of the entire data set (each input node has a local dataset of size q J ), p = L J+1 the size of the input layer of NN (J + 1) and s the size in bits of a parameter. Since as per (3.4), the output of the last layers of the input NNs are concatenated at the input of NN (J + 1) whose size is p, and each activation value is s bits, one then needs 2sp J bits for each data point -the factor 2 accounts for both the forward and backward passes; and, so, for an epoch our in-network learning requires 2pqs J bits. Note that the bandwidth requirement of in-network learning does not depend on the sizes of the NNs used at the various nodes, but does depend on the size of the dataset. For comparison, notice that with FL one would require 2NJs, where N designates the number of (weight and bias) parameters of an NN at one node. For SL, assuming for simplicity that the NNs j = 1, . . . , J all have the same size ηN, where η ∈ [0, 1], SL requires (2pq + ηNJ)s bits for an entire epoch. The bandwidth requirements of the three schemes are summarised and compared in Table 3.1 for two popular neural networks, VGG16 (N = 138, 344, 128 parameters) and ResNet50 (N = 25, 636, 712 parameters) and two example datsets, q = 50, 000 data points and q = 500, 000 data points. The numerical values are set as J = 500, p = 25088 and η = 0.88 for ResNet50 and 0.11 for VGG16.

Experiments

We perform two series of experiments. In both cases, the used dataset is the CIFAR-10

and

Experiment 1

For FL, each of the five client nodes is equipped with the entire network of Figure 3.4.

The dataset is split into five sets of equal sizes; and the split is now performed such that all five noisy versions of a same CIFAR-10 image are presented to the same client NN (distinct clients observe different images, however). For SL of [START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF], each input node is equipped with an NN formed by all fives branches with convolution networks (i.e., all the network of Fig.

Derivation of In-network Learning Loss Function

As mention in the introduction in this section, we restate the derivation of the loss function (3.5) presented in Aguerri-Zaidi [START_REF] Aguerri | Distributed variational representation learning[END_REF] for completeness. The review is brief and for more details please see [START_REF] Aguerri | Distributed variational representation learning[END_REF]. The distributed inference problem presented is related to the J-encoder CEO problem under log-loss presented in [START_REF] Courtade | Multiterminal source coding with an entropybased distortion measure[END_REF]. One of the main differences is that in the CEO problem setting n samples are compressed and transmitted jointly, as opposed to just one sample as is the case in an inference problem.

We first introduce the CEO problem, and then we present how known results for the CEO problem were used by [START_REF] Aguerri | Distributed variational representation learning[END_REF] to derive the loss function for the distributed training and inference problem given by (3.5). Y n , taking values in Y n , which is related to the observed random variables X n j , taking values in X n j ,through the joint probability mass function P X n J ,Y n := n i=1 P X J ,Y . We additionally assume that for all S ⊆ J we have X S -Y -X S c , i.e. the data observed by each encoder is independent of the rest of the observation, given Y. Each encoder j ∈ J can use nR j bits to encode the observed data X n j . We choose the reconstruction set Ŷn to be the set of distributions on Y n , i.e., Ŷn = P(Y n ); and we measure the discrepancies between the true values of Y n ∈ Y n and their estimated fits in terms of their per sample average logarithmic loss, i.e., for (y n , ŷn )

CEO Problem Setup

∈ Y n × P(Y n ) d (n) (y n , ŷn ) = 1 n n i=1 log 1 ŷi (y i ) . ( 3
ϕ n j : X n j -→ [1 : 2 nR j ]; (3.11)
and the decoding function at node (J + 1)

ψ : [1 : 2 nR 1 ] × . . . × [1 : 2 nR J ] -→ Ŷn . (3.12)
Definition 3.4.2. A rate-distortion tuple (R 1 , . . . , R J , D) is achievable if there exist a block length n, and encoding and decoding functions such that

R j ≥ 1 n log M (n) j , for j ∈ J, (3.13)
D ≤ E d (n) (Y n , ψ n (ϕ n 1 (X n 1 ), . . . ϕ n J (X n J ))) (3.14)
The rate-distortion region of the defined CEO problem, denoted as RI CEO , is defined as the closure of all non-negative (R 1 , . . . , R J , D) rate-distortion tuples that are achievable.

The following theorem characterised the rate-distortion region of the introduced CEO problem, the region was first characterised in [START_REF] Courtade | Multiterminal source coding with an entropybased distortion measure[END_REF].

Theorem 3.4.1. [28, Theorem 10] The rate-distortion region of the described CEO problem is given by the union of all tuples (R 1 , . . . , R J , D) satisfying for all S ⊆ J

j∈S R j + D ≤ j∈S I(X j ; U j |Y, Q) + H(Y|U S c , Q) (3.15)
for some set of pmfs P := {P U 1 |X 1 ,Q , . . . , P U J |X J ,Q , P Q } with the joint distribution of the form p(y)p(q) J j=1 p(x j |y)p(u j |x J , q).

Distributed Information Bottleneck

The authors of [START_REF] Aguerri | Distributed variational representation learning[END_REF] consider the problem in which the logarithmic distortion of the presented CEO problem is replaced by the mutual information constraint

∆ ≤ 1 n I Y n , ψ n (ϕ n 1 (X n 1 ), . . . ϕ n J (X n J )) (3.16)
where ∆ denotes the relevance of the model. Similar to the CEO problem, one now wants to find the set of achivable relevance-complexity tuples (R 1 , . . . , R J , ∆), this can be seen as a generalisation of the IB problem [START_REF] Tishby | The information bottleneck method[END_REF] to the distributed case.

Definition 3.4.3.

A relevance complexity tuple (R 1 , . . . , R J , ∆) is achievable if there exist a block length n, and encoding and decoding functions such that

R j ≥ 1 n log M (n) j , for j ∈ J, (3.17)
∆ ≤ 1 n I(Y n , ψ n (ϕ n 1 (X n 1 ), . . . ϕ n J (X n J ))) (3.18)
The relevance complexity region, denoted as RI DIB is defined as the closure of all non-negative (R 1 , . . . , R J , ∆) relavance-complexity tuples that are achievable.

By showing the equivalence between the two regions, RI DIB and RI CEO , the authors of [START_REF] Aguerri | Distributed variational representation learning[END_REF] are able to characterise the RI DIB region as follows Theorem 3.4.2. [10, Theorem 1] The relevance-complexity region, denoted as RI DIB , is given by the union of all tuples (R 1 , . . . , R J , ∆) satisfying for all S ⊆ J j∈S

R j ≤ ∆ + j∈S I(X j ; U j |Y, Q) + I(Y; U S c , Q) (3.19)
for some set of pmfs P := {P U 1 |X 1 ,Q , . . . , P U J |X J ,Q , P Q } with the joint distribution of the form p(y)p(q) J j=1 p(x j |y)p(u j |x J , q).

For simplicity, they consider the region of achievable relevance-complexity tuples under sum-complexity constrained, defined by

RI sum DIB := {(∆, R sum )∃(R 1 , . . . , R J )s.t.(∆, R 1 , R J ) ∈ RIs DIB and R sum = J j=1 R j } (3.20)
In practice, we are interested in finding the pmfs

P := {P U 1 |X 1 ,Q , .
R s := I(Y; U * J ) + J j=1 I(X j ; U * j ) -I(Y; U * j )] (3.22)
and P * is the set of pmfs that maximize the cost function

L DIB s = -H(Y|U J ) -s J j=1
H(Y|U j ) + I(U j ; X j ) .

(3.23)

It can be seen from (3.23) that, similar to the IB problem, by controlling s one can control the trade-off between the sum complexity of the latent representation U J and the relevance of the model. Unfortunately, optimising (3.23) requires computing the marginal distributions P Y|U 1 ,...,U J ,P Y|U 1 ,. . . ,P Y|U J , which is not always possible. To overcome this issue, one can introduce the variational distributions

Q := Q Y|U 1 , Q . . . , QY|U J , Q Y|U 1 ,...,U J , Q U 1 , . . . , Q U J (3.24)
and the variation cost function given by to the encoding distribution P and variations distribution Q which can be parameterized by neural networks as described in Section 2.3.2. More specifically, let P θ j (U j |X j ) be the encoding distribution at node j ∈ J parameterized by an NN with parameters θ j .

L VDIB s (P, Q) := E[log Q Y|U J (Y|U J )] + s J j=1 E[log Q Y|U j (Y|U j )] -D KL (P U j |X j ||Q U j ) (3.
Additionally, let log Q ϕ j (Y|U j ),j ∈ J, be the decoding mapping from U j to Y given by an NN with parameters ϕ j and let log Q ϕ J (Y|U 1 , . . . , U J ) be the main decoding distribution

given by an NN with parameters ϕ J . Finally, let Q φ j (U j ),j ∈ J, be some fixed prior over the latent space. By imposing such a restriction, the maximisation problem of where θ = [θ 1 , . . . , θ J ],φ = [φ 1 , . . . , φ J ],ϕ = [ϕ 1 , . . . , ϕ J , ϕ J ] and

L NNe s := E[log Q ϕ J (Y|U 1 , . . . , U J )] + s J j=1 E[log Q ϕ j (y i |u j,i )] -D KL P θ j (U j |X j )||Q φ j (U j ) . (3.29)
The loss function (3.5) can be obtained by taking the empirical estimate of (3.29).

Chapter 4

In-network Learning: Graph Network

Topology

In this chapter we discuss the extension of the INL to network topologies that can be modelled with a directed, acyclic graph. This problem was proposed and studied in the patent document [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF]. In this chapter, we derive a suitable loss function for such a model, by extending the techniques from [START_REF] Aguerri | Distributed variational representation learning[END_REF] to this more general case. More specifically, we consider the case in which the J nodes that observe the data do not communicate directly with the central node. Instead, the nodes are connected to the central node through intermediary nodes, which do not have data of their own; however, they do have computational capabilities to process the data they observe.

Moreover, we assume that some of the nodes that observe the data can be connected to other nodes that observe the data. We first derive an achievable trade-off between the performance of the system under log-loss and the complexities of the encoded values that the nodes need to exchange, as measured by their minimum description length (MDL). Based on this theoretical analysis, one can derive a loss function that can train the NNs in a distributed manner, by first deriving a variational inner bound, and then parameterizing that lower/inner bound using neural networks. We showcase these steps for an example five-node network.

Problem Setup

We model an N-node network by a directed acyclic graph G = (N, E, C), where N = [1 :

N] is the set of nodes, E ⊂ N × N is the set of edges and C = {C jk : ( j, k) ∈ E} is the set of edge weights. Each node represents a device and each edge represents a noiseless communication link with capacity C jk . The processing at the nodes of the set J is such that each of them assigns an index m jl ∈ [1, M jl ] to each x j ∈ X j and each received index tuple (m i j : (i, j) ∈ E) for each edge (j, l) ∈ E. Specifically, let for j ∈ J and l such that ( j, l) ∈ E, the set M jl = [1 : M jl ]. The encoding function at node j is

ϕ j : X j ×          i : (i, j) ∈ E M i j          -→ l : ( j,l) ∈ E M jl , (4.1)
where designates the Cartesian product of sets. Similarly, for k ∈ [1 :

N -1]/J, node k assigns an index m kl ∈ [1, M kl ] to each index tuple (m ik : (i, k) ∈ E) for each edge (k, l) ∈ E. That is, ϕ k : i : (i,k) ∈ E M ik -→ l : (k,l) ∈ E M kl . (4.2)
The range of the encoding functions {ϕ i } are restricted in size, as

log |M i j | ≤ C i j ∀i ∈ [1, N -1] and ∀ j : (i, j) ∈ E. (4.3)
Node N needs to infer on the random variable Y ∈ Y using all incoming messages, i.e., ψ : As such, the performance of a distributed inference scheme (ϕ j ) j∈J , (ϕ k ) k∈[1,N-1]/J , ψ for which (4.3) is fulfilled is given by its achievable relevance, given by

i : (i,N) ∈ E M iN -→ Ŷ. ( 4
∆ = H(Y) -E d(Y, Ŷ) , (4.6)
which, for a discrete set Y, is directly related to the error of misclassifying the variable

Y ∈ Y.

In-network Learning Generalisation

In practice, in a supervised setting, the mappings given by (4.1), (4.2) and (4.4) need to be learnt from a set of training data samples {(x 1,i , . . . , x J,i , y i )} n i=1 . The data are distributed such that the samples x j := (x j,1 , . . . , x j,n ) are available at node j for j ∈ J and the desired predictions y := (y 1 , . . . , y n ) are available at the end decision node N. We parameterize the possibly stochastic mappings (4.1), (4.2) and (4.4) using NNs. This is depicted in Figure 4.2. We denote the parameters of the NNs that parameterize the encoding function at each node i ∈ [1 : (N -1)] with θ i and the parameters of the NN that parameterize the decoding function at node N with ϕ. Let θ = [θ 1 , . . . , θ N-1 ], we aim to find the parameters θ, ϕ that maximize the relevance of the network, given the network constraints of (4.3). Given that the actual distribution is unknown and we only have access to a dataset, the loss function must strike a balance between its performance on the data set, given by an empirical estimate of the relevance, and the network's ability to perform well on samples outside the dataset.

The NNs at the various nodes are arbitrary and can be chosen independentlyfor instance, they need not be identical as in FL. It is only required that the following mild condition which, as will become clearer from what follows, facilitates the back-propagation be met. Specifically, for every j ∈ J and x j ∈ X j1 it holds that Size of first layer of NN (j) = Dimension (x j ) + i : (i,j) ∈ E (Size of last layer of NN (i)). where the maximisation is over joint measures of the form P Q P X 1 ,...,X J ,Y J j=1 P U j |X j ,Q , for which there exist non-negative R 1 , . . . , R J that satisfy with S ∩ J ∅. (4.12)

Proof. The proof of Theorem 4.2.1 appears in Section 4.4.1. An outline is as follows.

The result is achieved using a separate compression-transmission-estimation scheme in which the observations (x 1 , . . . , x J ) are first compressed distributively using Berger-Tung coding [START_REF] Berger | Multiterminal source encoding with one distortion criterion[END_REF] into representations (u 1 , . . . , u J ); and, then, the bin indices are transmitted as independent messages over the network G using linear-network coding [25, Section

Example: Five Node Network

Part of the utility of the loss function of Theorem 4.2.1 is in that it accounts explicitly for the topology of the network for inference fusion and propagation. Also, although as seen from its proof the setting of Theorem 4.2.1 assumes knowledge of the joint distribution of the tuple (X 1 , . . . , X J , Y), the result can be used to train, distributively, NNs from a set of available datesets. To do so, we first derive a Lagrangian function, from Theorem 4.2.1, which can be used as an objective function to find the desired set of encoders and decoder. Afterwards, we use a variational approximation to avoid the computation of marginal distributions, which can be costly in practice. Finally, we parameterize the distributions using NNs. For a given network topology in essence, the approach generalises that of the previous chapter to more general networks that involve hops. For simplicity, in what follows, this is illustrated for the example architecture of

∆ ≤ I (Y; U 1 , U 2 , U 3 ) (4.13a) C sum ≥ I(X 1 , X 2 , X 3 ; U 1 , U 2 , U 3 ) + I(X 2 , X 3 ; U 2 , U 3 |U 1 ) (4.13b)
for some measure of the form

P Y P X 1 ,X 2 ,X 3 |Y P U 1 |X 1 P U 2 |X 2 P U 3 |X 3 . ( 4
C s = I(X 1 , X 2 , X 3 ; U * 1 , U * 2 , U * 3 ) + I(X 2 , X 3 ; U * 2 , U * 3 |U * 1 ), (4.15b)
and P * is the set of pmfs

P := {P U 1 |X 1 , P U 2 |X 2 , P U 3 |X 3 } that maximise the cost function L s (P) := -H(Y|U 1 , U 2 , U 3 ) -sI(X 1 , X 2 , X 3 ; U 1 , U 2 , U 3 ) -sI(X 2 , X 3 ; U 2 , U 3 |U 1 ). ( 4
-(U 2 , U 3 ) -(X 1 , X 2 , X 3 , Y, U 1 )
. That is, the joint distribution factorises as

P X 1 ,X 2 ,X 3 ,Y,U 1 ,U 2 ,U 3 ,U 4 = P X 1 ,X 2 ,X 3 ,Y P U 1 |X 1 P U 2 |X 2 P U 3 |X 3 P U 4 |U 2 ,U 3 . (4.17)
Let for a given s ≥ 0 and conditional P U 4 |U 2 ,U 3 the Lagrange term

L low s (P, P U 4 |U 2 ,U 3 ) = -H(Y|U 1 , U 4 ) -sI(X 1 ; U 1 )
-2s I(X 2 ; U 2 ) + I(X 3 ; U 3 )

+ 2s I(U 2 ; U 1 ) + I(U 3 ; U 1 , U 2 ) . ( 4
Q := {Q Y|U 1 ,U 4 , Q U 3 , Q U 2 , Q U 1 }, (4.20)
where Q Y|U 1 ,U 4 represents variational (possibly stochastic) decoders and Q U 3 , Q U 2 and

Q U 1 represent priors. Also, let L v-low s (P + , Q) := E[log Q Y|U 1 ,U 4 (Y|U 1 , U 4 )] -sD KL (P U 1 |X 1 ∥Q U 1 ) -2sD KL (P U 2 |X 2 ∥Q U 2 ) -2sD KL (P U 3 |X 3 ∥Q U 3 ). (4.21)
The following lemma, the proof of which is essentially similar to that of [START_REF] Aguerri | Distributed variational representation learning[END_REF]Lemma 1], provides an alternative maximisation problem to that of max P + L low s (P + ), for every s ≥ 0. Since the distribution of the data is not known, but only a set of samples is available

{(x 1,i , . . . , x J,i , y i )} n i=1
, we restrict the optimisation of (4.21) to the family of distributions that can be parametrised by NNs. Thus, we obtain the following loss function which can be optimised empirically, in a distributed manner, using gradient-based techniques,

L NN s (n) := 1 n n i=1 log Q ϕ 5 (y i |u 1,i , u 4,i ) -s log P θ 1 (u 1,i |x 1,i ) Q φ 1 (u 1,i ) - 2s n n i=1 log P θ 2 (u 2,i |x 2,i ) Q φ 2 (u 2,i ) + log P θ 3 (u 3,i |x 3,i ) Q φ 3 (u 3,i ) , (4.23)
where s stands for a Lagrange multiplier and the distributions Q ϕ 5 , P θ 4 , P θ 3 , P θ 2 , P θ 1 are variational ones whose parameters are determined by the chosen NNs using the re-parametrisation trick of [START_REF] Kingma | Auto-encoding variational bayes[END_REF]; and, {Q φ i : i ∈ {1, 2, 3}} are priors known to the encoders. For example, denoting by f θ j the NN used at node j ∈ J whose (weight and bias) parameters are given by θ j . For regression problems the conditional distribution P θ j (u i, j |x i, j ) can be chosen to be multivariate Gaussian, i.e., P θ j (u i, j |x i, j ) = N(u i,j ; µ

θ j i , Σ θ j i
). For discrete data, concrete variables (i.e., Gumbel-Softmax) can be used instead.

Training Algorithm

We now describe the training algorithm for INL for the five-node example, this section particularises on the training procedure first reported in the patent [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] until the last layer of NN 5. All nodes update their parameters using standard backpropagation technique which is as follows. For node t ∈ N let L t denote the index of the last layer of the NN at node t. Also, let, for w [l] t , b [l] t and a [l] t denote, respectively, the weights, biases and activation values at layer l ∈ [2 : L t ] for the NN at node t; σ is the activation function and the input to the NN is denoted a [1] t . Node t computes the error vectors

δ [L t ] t = ∇ a [L t ] t L NN s (b) ⊙ σ ′ (w [L t ] t a [L t -1] t + b [L t ] t ) (4.24a) δ [l] t = [(w [l+1] t ) T δ [l+1] t ] ⊙ σ ′ (w [l] t a [l-1] t + b [l] t ) ∀ l ∈ [2, L t -1], (4.24b)
δ [1] t = [(w [2] t ) T δ [2] t ] (4.24c)

and then updates its weight-and bias parameters as

w [l] t → w [l] t -ηδ [l] t (a [l-1] t ) T , (4.25a) b [l] t → b [l] t -ηδ [l] t , (4.25b)
where η designates the learning parameter4 .

Inference Algorithm

We now describe the inference algorithm for INL for the five-node example, this section similarly particularises on the training procedure first reported in the patent [START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] by

Zaidi, Scaman and Escamilla to the five-node network presented. During inference, nodes 1, 2 and 3 observe (or measure) each a new sample. Let x 1 be the sample observed by node 1; and x 2 and x 3 those observed by node 2 and node 3, respectively.

Node 1 processes x 1 using its NN and sends an encoded value u 1 to node 5; and so do nodes 2 and 3 towards node 4. Upon receiving u 2 and u 3 from nodes 2 and 3, node 4 concatenates them vertically and processes the obtained vector using its NN.

The output u 4 is then sent to node 5. The latter performs similar operations on the activation values u 1 and u 4 ; and outputs an estimate of the label y in the form of a soft output Q ϕ 5 (y|u 1 , u 4 ).

Proofs

Proof of Theorem 4.2.1

The proof of Theorem 1 is based on a scheme in which the observations {x j } j∈J are compressed distributively using Berger-Tung coding [START_REF] Berger | Multiterminal source encoding with one distortion criterion[END_REF]; and, then, the compression bin indices are transmitted as independent messages over the network G using linear-network coding [START_REF] Gamal | Network Information Theory[END_REF]Section 15.4]. The decision node N first decompresses the compression codewords and then uses them to produce an estimate Ŷ of Y. In what follows, for simplicity, we set the time-sharing random variable to be a constant, i.e.,

Q = ∅. Let 0 < ϵ ′′ < ϵ ′ < ϵ.

Codebook Generation

Fix a joint distribution P X 1 ,...,X J ,Y,U 1 ,...,U J that is factorised as P Q P X 1 ,...,X J ,Y J j=1 P U j |X j ,Q . Also, let D = H(Y|U 1 , . . . , U J ); and, for (u 1 , . . . , u J ) ∈ U 1 × . . . × U J , the reconstruction func-

tion ŷ(•|u 1 , . . . , u J ) ∈ P(Y) such that E d(Y, Ŷ) ≤ D 1 + ϵ
, where d :

Y × P(Y) -→ R +
is the distortion measure given by (4.5). For every j ∈ J, let Rj ≥ R j . Also, randomly and independently generate 2 n Rj sequences u n j (l j ), l j ∈ [1 : 2 n Rj ], each ac-cording to n i=1 p U j (u ji ). Partition the set of indices l j ∈ 2 n Rj into equal size bins

B j (m j ) = (m j -1)2 n Rj -R j : m j 2 n Rj -R j , m j ∈ [1 : 2 nR j ].
The codebook is revealed to all source nodes j ∈ J as well as to the decision node N, but not to the intermediary nodes.

Compression of the observations

Node j ∈ J observes x n j and finds an index l j ∈ [1 :

2 n Rj ] such that (x n j , u n j (l j )) ∈ T (n) ϵ ′′ . If
there is more than one index the node selects one at random. If there is no such index, it selects one at random from [1 : 2 n Rj ]. Let m j be the index of the bin that contains the selected l j , i.e., l j ∈ B j (m j ).

Transmission of the compression indices over the graph network

In order to transmit the bins indices (M 1 , . . . , M J ) ∈ [1 : where C(S) is defined by (4.9).

2 nR 1 ] × . . . × [1 : 2 nR J ]

Decompression and estimation

The decision node N first looks for the unique tuple ( l1 , . . . , lJ ) ∈ B 1 (m 1 ) × . . . × B J (m J ) such that (u n 1 ( l1 ), . . . , u n J ( lJ )) ∈ T (n) ϵ . With high probability, Node N finds such a unique tuple as long as n is large and for all S ⊆ J it holds that [START_REF] Berger | Multiterminal source encoding with one distortion criterion[END_REF] (see also [25, Theorem

12.1]) j∈S R j ≥ I(U S ; X S |U S c ). (4.27)
The decision node N then produces an estimate ŷn of y n as ŷ(u n 1 ( l1 ), . . . , u n J ( lJ )).

It can be shown easily that the per-sample relevance level achieved using the described scheme is ∆ = I(U 1 , . . . , U J ; Y); and this completes the proof of Theorem 4.2.1.

Proof of Proposition 4.3.1

For C sum ≥ 0 fix s ≥ 0 such that C s = C sum ; and let

P * = {P U * 1 |X 1 , P U * 2 |X 2 , P U * 3 |X
3 } be the solution to (4.16) for the given s. By making the substitution in (4.15):

∆ s =I(Y; U * 1 , U * 2 , U * 3 ) (4.28) ≤∆ (4.29)
where (4.29) holds since ∆ is the maximum I(Y; U 1 , U 2 , U 3 ) over all distribution for which (4.13b) holds, which includes P * .

Conversely let P * be such that (∆, C sum ) is on the bound of the RI sum then:

∆ =H(Y) -H(Y|U * 1 , U * 2 , U * 3 ) ≤H(Y) -H(Y|U * 1 , U * 2 , U * 3 ) + sC sum -s I(X 2 , X 3 ; U * 2 , U * 3 |U * 1 ) + I(X 1 , X 2 , X 3 ; U * 1 , U * 2 , U * 3 ) (4.30)
≤H(Y) + max P L s (P) + sC sum (4.31)

=∆ s -sC s + sC sum =∆ s + s(C sum -C s ). (4.32)
Where (4.30) follows from (4.13b). Inequality (4.31) holds due to the fact that max P L(P)

takes place over all P, including P * . Since (4.32) is true for any s ≥ 0 we take s such that

C sum = C s , which implies ∆ ≤ ∆ s .
Together with (4.29) this completes the proof.

Proof of Lemma 4.3.1

We have

L s (P) = -H(Y|U 1 , U 2 , U 3 ) -sI(X 1 , X 2 , X 3 ; U 1 , U 2 , U 3 ) -sI(X 2 , X 3 ; U 2 , U 3 |U 1 ) (4.33) = -H(Y|U 1 , U 2 , U 3 ) -s I(X 1 ; U 1 ) + 2I(X 2 , X 3 ; U 2 , U 3 |U 1 ) (4.34) = -H(Y|U 1 , U 2 , U 3 ) -sI(X 1 ; U 1 ) -2sI(X 2 ; U 2 ) -2s I(X 3 ; U 3 ) -I(U 3 ; U 1 , U 2 ) -I(U 2 ; U 1 ) (4.35)
= -H(Y|U 1 , U 2 , U 3 ) -sI(X 1 ; U 1 ) -2sI(X 2 ; U 2 )

+ 2s I(U 2 ; U 1 ) + I(U 3 ; U 1 , U 2 ) -I(X 3 ; U 3 ) (4.36)
≥ -H(Y|U 1 , U 4 ) -sI(X 1 ; U 1 ) -2s I(X 2 ; U 2 ) + I(X 3 ; U 3 )

+ 2s I(U 2 ; U 1 ) + I(U 3 ; U 1 , U 2 ) (4.37)
where (4.34) holds since

U 1 -X 1 -(X 2 , X 3 , U 2 , U 3 ) and (U 2 , U 3 ) -(X 2 , X 3 ) -(U 1 , X 1 ); (4.35) holds since U 2 -X 2 -(U 1 , X 3 ) and U 3 -X 3 -(U 1 , U 2 , X 2 ); (4.37) hold since U 4 -(U 2 , U 3 ) - (Y, U 1 ).

Proof of Lemma 4.3.2

From [10, eq. ( 55)] it can be shown that for any pmf Q Y|Z (y|z) , y ∈ Y and z ∈ Z the conditional entropy H(Y|Z) is :

H(Y|Z) = E[-log Q Y|Z (Y|Z)] -D KL (P Y|Z ||Q Y|Z ). (4.38)
And from [10, eq. ( 81)]:

I(X; Z) = H(Z) -H(Z|X) = D KL (P Z|X ∥Q Z ) -D KL (P Z ∥Q Z ). (4.39)
Now, substituting equations (4.38) and (4.39) in (4.21) the following result is obtained:

L low s (P + ) = -H(Y|U 1 , U 4 ) + 2s I(U 2 ; U 1 ) + I(U 3 ; U 1 , U 2 ) -I(X 2 ; U 2 ) -I(X 3 ; U 3 ) =E[log Q Y|U 1 ,U 4 ] + D KL (P Y|U 1 ,U 4 ||Q Y|U 1 ,U 4 ) -sD KL (P U 1 |X 1 ∥Q U 1 ) + sD KL (P U 1 ∥Q U 1 ) -2s D KL (P U 2 |X 2 ∥Q U 2 ) + D KL (P U 3 |X 3 ∥Q U 3 ) + D KL (P U 2 ∥Q U 2 ) + D KL (P U 3 ∥Q U 3 ) + 2s D KL (P U 2 ∥Q U 2 ) + D KL (P U 3 |U 1 ,U 2 ∥Q U 3 ) + D KL (P U 3 ∥Q U 3 ) + D KL (P U 2 |U 1 ∥Q U 2 ) =E[log Q Y|U 1 ,U 4 ] -sD KL (P U 1 |X 1 ∥Q U 1 ) -2sD KL (P U 2 |X 2 ∥Q U 2 ) -2sD KL (P U 3 |X 3 ∥Q U 3 ) + sD KL (P U 1 ∥Q U 1 ) + 2sD KL (P U 2 |U 1 ∥Q U 2 ) + 2sD KL (P U 3 |U 1 ,U 2 ∥Q U 3 ) + D KL (P Y|U 1 ,U 4 ||Q Y|U 1 ,U 4 ) =L v-low s + sD KL (P U 1 ∥Q U 1 ) + 2sD KL (P U 2 |U 1 ∥Q U 2 ) + 2sD KL (P U 3 |U 1 ,U 2 ∥Q U 3 ) + D KL (P Y|U 1 ,U 4 ||Q Y|U 1 ,U 4 ) ≥L v-low s (4.40)
The last inequality (4.40) holds due to the fact that KL divergence is always positive and s ≥ 0, thus proving the lemma.

of Y based only on compressed versions of the devices' observations. We are interested in the setting in which all devices, including the FC, are equipped with neural networks (NN) and use them to perform some desired operations. We consider the case in which the available communication resources can be arbitrarily allocated between nodes with the aim of improving the quality of the estimate Ŷ of Y, on average. To accurately assign the resources, one needs to be able to correctly assess the benefits of allocating resources to each node, which cannot be computed directly due to the distributed nature of the network. It is interesting to observe that while for the inference task at hand, the FC has a clear advantage over each of the devices taken individually, as it observes the compressed representations, which we denote by U j for j ∈ J = {1, . . . , J}, from all the nodes. The latter have, in their turn, an advantage over the FC. Indeed, rather than a compressed version of the input data, each device has access to the data that has been used to generate that compression. According to the Markov chain Y -X j -U j , it is clear that U j holds less information about Y than X j itself, for any j ∈ J. Consequently, relevant network resource allocation procedures for such inference tasks look to suitably strike the right balance between the aforementioned advantage. In [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF], the authors patented an architecture in which (i) each device measuring its own local assessment of how relevant is the data it holds for the inference task at hand and transmitting it to the decision node, (ii) that device also simultaneously sends the output activation values to the decision node, (iii) the decision node combining all received information (activation values from the various devices as well as their individual local assessments) and then forming its own (global) assessment of how relevant each device's observation is, (iv) application of the idea to device scheduling and power allocation (two algorithms for scheduling were already given in [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF]). This architecture was described in detail in [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF].

In this chapter, we build upon these works by discussing possible local assessments as well as introducing a novel global assessment measure. Additionally, we provide a new scheduling algorithm that does not require the channel knowledge, as well as, a new power allocation algorithm. A similar work on the development of resource allocation algorithms based on the patent of [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF] had been started in [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]; however, in [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF] the authors considered the problem of digital transmission and as such proposed algorithms for time slots allocation, as opposed to scheduling and power allocation.

Another key difference between our work and that of [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF] is that the global assessment is calculated differently.

Related Work

To the best of the authors' knowledge, the current resource allocation algorithms are not suitable for use for the inference problem studied in this chapter. This setting introduces the additional challenge of assessing the relative importance of each node in a distributed manner since each node is equipped with a NN, and the relationship between the observed data and the target variable must be learnt from a set of training data.

The works of [START_REF] Liu | Data-importance aware user scheduling for communication-efficient edge machine learning[END_REF][START_REF] Amiri | Update aware device scheduling for federated learning at the wireless edge[END_REF][START_REF] Shi | Joint device scheduling and resource allocation for latency constrained wireless federated learning[END_REF][START_REF] Perazzone | Communication-efficient device scheduling for federated learning using stochastic optimization[END_REF] consider the problem of resource allocation or scheduling in a distributed network when there is no known model of the relation between the data and the target variable, and the authors employ NNs as part of their solutions. However, these works consider device scheduling and resource allocation during distributed training, not distributed inference as in our case. Additionally, the model obtained from the training procedure is designed for use in a centralised manner. In [START_REF] Liu | Data-importance aware user scheduling for communication-efficient edge machine learning[END_REF], the authors looked at the problem of scheduling multiple edge devices that have access to local data that they need to send to the central node, over a noisy channel, such that it can train a central model. In [START_REF] Amiri | Update aware device scheduling for federated learning at the wireless edge[END_REF][START_REF] Shi | Joint device scheduling and resource allocation for latency constrained wireless federated learning[END_REF][START_REF] Perazzone | Communication-efficient device scheduling for federated learning using stochastic optimization[END_REF], the authors investigate the problem of power allocation and scheduling during the training procedure of the federated learning (FL) algorithm, introduced in [START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF] and presented in Section 3.3.1.

Our work is also related to the problem of distributed detection and distributed estimation. The problem of resource allocation for distributed detection has been extensively investigated and, as such, we refer the reader to [START_REF] Chamberland | Wireless sensors in distributed detection applications[END_REF] and references within for a more in depth survey of the field. Two papers that we consider most related to our work in this field are those of [START_REF] Wimalajeewa | Optimal power scheduling for correlated data fusion in wireless sensor networks via constrained pso[END_REF][START_REF] Zhang | Optimal power allocation for distributed detection over mimo channels in wireless sensor networks[END_REF]. In [START_REF] Zhang | Optimal power allocation for distributed detection over mimo channels in wireless sensor networks[END_REF], the authors derived the optimal power allocation, with respect to a so-called J-divergence performance index, in a WSN for the case where the observations are independent, but do not need to be sampled from identical distributions. Furthermore, the authors have shown that the optimal power allocation depends on both the quality of the observed data and on the quality of the communication channel. In [START_REF] Wimalajeewa | Optimal power scheduling for correlated data fusion in wireless sensor networks via constrained pso[END_REF], the authors used a particle swarm optimisation algorithm to find the power allocation for the distributed detection problem for the case of correlated observations. The authors observed that the performance of the system can be significantly improved by taking into account the correlation of the observed variables compared to assuming that they are independent. In both [START_REF] Wimalajeewa | Optimal power scheduling for correlated data fusion in wireless sensor networks via constrained pso[END_REF] and [START_REF] Zhang | Optimal power allocation for distributed detection over mimo channels in wireless sensor networks[END_REF], it is noted that, for optimal performance, nodes with poor observations and/or channel quality need to be turned off, i.e., not transmit data.

For the problem of distributed estimation, in [START_REF] Xiao | Decentralized estimation in an inhomogeneous sensing environment[END_REF][START_REF] Luo | An isotropic universal decentralized estimation scheme for a bandwidth constrained ad hoc sensor network[END_REF][START_REF] Luo | Universal decentralized estimation in a bandwidth constrained sensor network[END_REF], the authors investigated decentralised estimation schemes, with perfect channels, in which local nodes send to the FC messages of length determined by the signal-to-noise ratio (SNR) of the observed signal. In [START_REF] Xiao | Power scheduling of universal decentralized estimation in sensor networks[END_REF], the authors extended the results of [START_REF] Xiao | Decentralized estimation in an inhomogeneous sensing environment[END_REF] to the case of an imperfect communication channel and observed that, similarly to distributed detection, nodes with bad data quality or poor channels should be turned off.

As already described in the introduction, this chapter builds upon [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF], where the authors patented an architecture in which (i) each device measuring its own local assessment of how relevant is the data it holds for the inference task at hand and transmitting it to the decision node, (ii) that device also simultaneously sends the output activation values to the decision node, (iii) the decision node combining all received information (activation values from the various devices as well as their individual local assessments) and then forming its own (global) assessment of how relevant each device's observation is, (iv) application of the idea to device scheduling and power allocation (two algorithms for scheduling were already given in [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF]). This architecture was described in detail in [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]; however in [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF] the authors consider the problem of digital transmission and as such they propose algorithms for time slots allocation, as opposed to scheduling and power allocation which is considered in this chapter. In this chapter, we build upon these works by discussing possible local assessments as well as introducing a novel global assessment measure. Additionally, we provide a new scheduling algorithm that does not require the channel knowledge, as well as, a new power allocation algorithm. Consider the network inference problem of Figure 5.2. The problem setting is similar to that considered in Section 3, with the addition that the quality of the channels between Y and the local (edge) devices, and between the edge devices and the FC vary in a block-stationary manner. This can happen in WSNs, for example, due to faulty sensors or changes in the environment in which the WSN is deployed. We consider a block to be the period in which both the communication channels and the channels between Y and the local (edge) devices are stationary. For a given block, the goal is to schedule only a subset K of devices (those whose data and channels are comparably the best, in a sense that will become clearer in the rest of this chapter) and allocate power to the scheduled nodes such that the performance of the system is improved on average.

Resource Allocation and Scheduling Problem

The reason for selecting only a subset of devices comes from the fact that in doing so, we can allocate more power to the more relevant nodes. In our setting, the assessment of the usefulness of the data observed by a node depends not only on the inference task at hand but also on what other nodes observe.

Due to the fact that the relationship between the data observed by each node is In the rest of this chapter, we concentrate on the design of our solution for a given block. During both the scheduling phase and the inference phase, we assume digital transmission, and that the devices transmit over orthogonal channels, i.e., they do not interfere with each other. The channel gain from a sensing node j ∈ J to the FC is denoted by h j ∈ R + . The signals received by the FC are distorted by additive independent Gaussian noise, which we assume to have zero mean and unit variance.

The channel capacity between node j and the FC is given by

C j := 0.5 log 2 (1 + h 2 j P j ), (5.1)
where P j is the signal power at node j. For simplicity of analysis, we assume that the sensing nodes use capacity-achieving channel codes in order to reliably convey information to the FC. As a consequence, the jth sensing node can transmit up to C j bits of information to the FC. We assume that the channel gain is constant between the scheduling phase and the inference phase.

In this chapter, we focus on the scheduling and resource allocation problem, and do not discuss the training procedure. We assume that the network can be re-trained during the scheduling phase. We consider power to be the resource that needs to be allocated. We do not treat the problem of intelligent bandwidth allocation and assume that the bandwidth is allocated equally among the nodes during both the scheduling and inference phases. Additionally, we assume that the effect of the bandwidth on the quality of the channel is negligible. As such, the channel capacity is known and fixed for both the scheduling and inference phases. In the following section, we describe the scheduling and power allocation algorithms.

Proposed Architecture

Let B represent the total available frequency bandwidth of the communication channels between the nodes and the FC, and P represent the total available power to be allocated between the nodes. During the scheduling phase, J available subcarriers are used, with the bandwidth B allocated equally among them. Additionally, the total available power is also equally allocated among the nodes. The choice of equal power and bandwidth allocation is important, as it gives each node an equal chance to be selected during the scheduling phase. We divide the scheduling phase in two parts. During the first part, every device obtains its own measure of the relevancy of its observed data. We refer to this measure as the local assessment, and denote the value obtained with RI j for each node j ∈ J. See Section 5.3.1 for more information on how each node computes this local assessment. The purpose of these local assessments is to help the FC compensate for not observing X 1 , . . . , X J directly.

Each node j ∈ J transmits the obtained RI j to the FC. The transmission occurs over a negligible portion, αB/J, of the available bandwidth of the communication channel, with α ∈ [0 : 1] and close to 0. The rest of the bandwidth is allocated for the transmission of the encodings of x j , u j = [u j,1 , . . . , u j,n ]. This is shown in Figure 5.4a.

The second part of the scheduling phase begins once the FC receives all local assessments RI 1 , . . . , RI J as well as all standard features u 1 , . . . , u J . Using all the received data, the FC performs its own, global assessment of how relevant each node is. The procedure for computing this global assessment is detailed in Section 5.3.2. Then, the regarding the target variable should receive a higher value from the local assessment.

Finally, the local assessment should always output a positive value.

Global Assessment

The global assessment is a task-dependent value, assigned to each node, which indicates the FC's believed usefulness of that node, for the desired task. We denote such a value as a global relevance indicator. When the FC computes its believed usefulness of each node, it needs to take into account the information that the other nodes possess. It was already shown in [START_REF] Wimalajeewa | Optimal power scheduling for correlated data fusion in wireless sensor networks via constrained pso[END_REF] that power allocation leads to better performance when the correlation between the observations is taken into account, compared to assuming that the observations are independent.

Unfortunately, the FC only observes compressed versions of the data sensed by each node, for a given resource allocation, and not the data itself. As a consequence, the correlation between the data observed by each node cannot be computed. To overcome this issue, our proposed measure, the global relevance indicator, takes into account the correlation between the compressed representations that the FC receives, while also taking into account local assessments of the observed data. More specifically, the global relevance indicator is formed by combining

• the local assessments, described in Section 5.3.1, obtained by each node,

• assessments of the relative usefulness of the data received from each of the nodes.

We measure the relative usefulness of the data received from each node using a scaled conditional mutual information (CMI), i.e., to compute the usefulness of the data from node j, relative to node i, we compute I(Y; U j |U i ), and then scale it by I(Y; U 1 , . . . , U J ).

More specifically, for the case where there are J nodes, the global relevance indicator of a node j ∈ J, in relation to a subset of nodes K p ⊆ J/{ j}, is given by

GRI K p j = RI j J i=1 RI i I(Y; U j |U K p ) I(Y; U 1 , . . . , U J ) . ( 5
I(Y; U j |U K p ) = H(Y|U K p ) -H(Y|U K p ∪{j} ).
(5.4)

Each of the conditional entropies can be estimated using the decoding NN at the FC

H(Y|U K p ) ≈ n i=1 log Q ϕ (y i |u K p ,i ), (5.5)
where n is the number of samples available and ϕ are the parameters of the NN at the FC.

The NN at the decoder can be trained to estimate Y based on data from a variable number of nodes, e.g., using techniques to deal with missing data; see [START_REF] Little | Statistical analysis with missing data[END_REF] for more details, or it can be retrained for different combinations of available encoders. It is important to note that the estimator presented in (5.5) can be biased due to the variational approximation. Despite this, by using the same decoder as the one that will be used during inference, we will obtain a good measure of the usefulness of the nodes during the inference phase, as the bias will also be present during the inference phase.

A similar approach can be used to estimate the MI I(Y; U 1 , . . . , U J ). One can similarly split the term into a difference between two entropy terms

I(Y; U 1 , . . . , U J ) = H(Y) -H(Y|U 1 , . . . , U J ), (5.6)
and then estimate H(Y|U 1 , . . . , U J ) using (5.5) and estimate H(Y) using an empirical

Scheduling

The scheduling policy looks at selecting K nodes, of the total J nodes, to transmit during inference. As such, we use the previously defined global relevance indicator; and the FC selects the K nodes that are ranked the highest using the global relevance indicator technique.

More specifically, the FC ranks the nodes using the global relevance indicator as follows

r i = argmax j∈J/{r 1 ,...,r i-1 } GRI {r 1 ,...,r i-1 } j , (5.7)
where GRI {r 1 ,...,r i-1 } j is defined as in (5.3) and r i is the index of the node of rank i according to the global relevance indicator. The FC then selects the top K ranked nodes, we denote the set of such nodes as K = {r 1 , . . . , r K }. Given that in practice GRI {r 1 ,...,r i-1 } j cannot be computed, we instead use the estimate presented in (5.3), and denote it GRI . This process is described in Algorithm 1.

Algorithm 1: Algorithm 1 -estimate the relative importance coefficients Require: descriptions u j and local assessments RI j for j ∈ J, and the desired number of nodes K. Ensure: Rankings r 1 , . . . , r K and scores GRI r 1 , . . . , GRI r K .

1: initialize 2: r 0 = ∅ 3: for i = 1 to K do 4:

Compute GRI {r 0 ,...,r i-1 } j for every j ∈ J/{r 0 , . . . , r i-1 } using (5.3).

5:

Select node with highest GRI Save the computed global relevance indicator value GRI r i = GRI {r 0 ,...,r i-1 } r i 7: end for 8: return r 1 , . . . , r K , GRI r 1 , . . . , GRI r K .

Power Allocation

Once the subset K = {r 1 , . . . , r K } of nodes is selected, one needs to allocate the total available power to them. The simplest way would be to allocate the power equally among the nodes; however, this would not be an efficient way of power allocation. The waterfilling [START_REF] Telatar | Capacity of multi-antenna gaussian channels[END_REF][START_REF] Thomas | Elements of information theory[END_REF] approach has shown that to maximise the amount of information that can be transmitted, it is better to allocate more power to the nodes with better channels. At the same time, each node is limited by the amount of useful information it observes regarding the variable of interest Y; any power allowing the node to send more information than what it observes about Y will not be able to improve the performance of the network. Finally, the power allocation also needs to take into account the usefulness of the node for inferring Y at the FC. As such, the power allocation algorithm needs to strike a balance between maximizing the total information the nodes transmit to the FC, the usefulness of the transmitted information towards inferring Y, and the total amount of useful information regarding Y that each node can transmit to the FC during inference.

To achieve this desired trade-off, we introduce a measure, that for each node r k ∈ K , combines the usefulness of the node towards inferring Y under equal power allocation, and the amount of information regarding Y that is contained in X r k but could not be transmitted, under equal power allocation. We denote this term as the expected usefulness (EU), and define it, for the node ranked in r k as

EU r k = I(Y; U r k |U r 1 , . . . , U r k-1 ) I(Y; X r k |U r k ) I(Y; X r k ) (5.8)
The first term, I(Y; U r k |U r 1 , . . . , U r k-1 ), is estimated during the scheduling phase and quantifies the usefulness of the node under equal power allocation. Although not directly related to the channel gain, it can be observed that, under equal power allocation, a worse channel would be able to transmit less information compared to a node with better channel, leading to a smaller I(Y; U r k |U r 1 , . . . , U r k-1 ), everything else being the same. As such, nodes with better channels will tend to have higher EU which will lead to higher power allocation, as will be seen later. This ensures that the intuition behind the waterfilling approach is met. On the other hand, we also need to ensure that the node observes enough information about Y such that it can make use of the allocated power. To resolve this, the second term, I(Y; X r k |U r k ) I(Y; X r k ) , is introduced, to estimate the amount of information the node was not able to transmit, regarding Y, due to the channel constraints. The term can be rewritten as

I(Y; X r k |U r k ) I(Y; X r k ) = I(Y; X r k ) -I(Y; U r k ) I(Y; X r k ) (5.9)
where the term I(Y; X r k ) = RI r k as described in Section 5. Afterwards, the total power, denoted by P, is distributed proportionally to the assigned EU. More specifically, the power allocated to node r k ∈ K , which we denote by P r k , is computed as

P r k = EU r k r j ∈K EU r j P ∀r k ∈ K . (5.10)
Remark 5.3.1. The power allocation strategy described by (5.10) allocates the total power to the selected nodes. The same technique can also be used to allocate only the power released by removing or shutting off some of the nodes. Effectively (5.10) can be changed to

P r k = P J + EU r k r j ∈K EU r j P(J -K) J ∀r k ∈ K . (5.11)
Using (5.11) can have an advantage, compared to (5.10), as it will prevent extreme cases, in which most of the power goes to only one node. The disadvantage is that the improvement over equal power allocation will be reduced.

Experiments

We perform a series of experiments to highlight the advantages of our technique. We showcase how the scheduling algorithm of Section 5.3.3 takes into account the channel gain, local assessment and the relationship between the data observed by the nodes when making the selection. Additionally, we showcase the improvements our power such that Û can be transmitted over a channel using R bits, with minimal mean square error distortion and in the large data regime, the optimal test channel p(U| Û) is given by

U = Û + Z with (5.15) Z ∼ N(0, σ 2 Z ); Û ∼ N(0, σ 2 U -σ 2 Z ); (5.16) R = 1 2 log 2 σ 2 U σ 2 Z .
(5.17)

To derive the noise variance for each node, we use equations (5.17), (5.12) and (5.1) from which we obtain

σ 2 Z j = σ 2 U j 1 + h 2 j P j . (5.18)
While equation (5.18) gives us the variance of the noise, it is important to note, from equation (5.15), that the noise is not independent of latent variable, i.e., Z j ̸ ⊥ U j . As such we derive p(Z j |U j ) using equation (5.15) and the fact that U j , Ûj , and Z j are all Gaussian

p(Z j |U j = u j ) ∼ N σ 2 Z j σ 2 U j u j ; σ 2 Z j - σ 4 Z j σ 2 U j . ( 5.19)
We then obtain equation (5.14) from the assumption that U j ∼ N(0, 1) ∀j ∈ J, and equations (5.19) and (5.18).

In our experiments, the local assessments, RIs, are calculated using the estimated mutual information between the local data and the label, which is estimated using a locally trained NN.

Scheduling Experiment

Our aim is to schedule K = 3 devices of the nine total nodes, given a total power of P = 100. We use the scheduling algorithm presented in Algorithm 1, the results of which are shown in Table 5.3. The table contains the GRI j value obtained for each node, given the previously selected nodes, with the values in green representing the highest values, i.e., the chosen nodes. The networks are re-trained for each configuration of the selected encoders for an accurate estimation of the entropy terms. The entropy of Y is estimated using the naive plug-in estimator [START_REF] Paninski | Estimation of entropy and mutual information[END_REF]. We remark that while the GRI j measure given by (5.2) is always positive, the empirical estimate GRI j computed using (5.3) can be negative for the case where (5.4) is very small due to estimation errors. a very bad channel, as such intuitively it makes sense to be selected. It is interesting to observe that node 3 is not selected. Since node 3 observes the same information as node 2, once node 2 is selected, the usefulness of node 3 drops significantly.

As observed in

The final node to be selected is node 7. It can be observed that the node has a balance of observing good quality data, compared to most of the remaining nodes, and a good channel. Additionally, node 7 observes data related to the lower part of the image, which neither node 2 or 4 do, and as such could provide complementary information.

This balance makes node 7 the most useful of the remaining nodes.

We compare our selection strategy with three other possible strategies. In the first case, we select the three nodes with the best channels. In the second case, we select the three nodes with the best local assessments. In the final case, we rank the nodes according to the CMI. More specifically, the node in rank i is selected using the following r i = arg max j∈J/{r 1 ,...,r i-1 } Î(Y; Ûj | Ûr 1 , . . . , Ûr i-1 ).

(5.20)

We then select the three nodes with the highest rank. This technique is similar to our approach, however, it does not incorporate the use of local assessments. The scheduling, however, changes the power allocation by allocating all the power to a subset of the nodes, the effect of which the FC cannot take into account, since it only observes compressed representations. By having each node compute a local assessment, which is then incorporated into the computation of the global relevance indicator, we mitigate the effect of the global assessment being computed only on compressed data. Using our proposed power allocation strategy, described in Section 5.3.4, node 2 is allocated 16.2% of the power, node 4 is allocated 62.6% of the power, while node 7 receives 21.2% of the remaining power. As can be seen from the allocation strategy, the algorithm looks to allocate power to the nodes that are both useful to the prediction of Y and also have additional information regarding Y that was not sent during the equal power allocation phase. Node 4, which both has a great impact on predicting Y and is unable to transmit all the useful data it observes, receives the highest amount of additional power. Node 2 and node 7 receive similar amounts of power. While node 2 is more useful towards predicting Y, node 7 has more additional information that was not transmitted during the equal power allocation phase. As can be seen in Figure 5.8, this power allocation strategy outperforms equal power allocation as well as equal power allocation when all the nodes are scheduled.

Power Allocation Experiment

Chapter 6

Conclusions and Perspectives

Conclusions

In this thesis, we have focused on the distributed training and inference problem. We discussed how multiple nodes can work together to jointly train a model that can be used in a distributed manner during the inference phase through a novel algorithm named in-network learning (INL). Compared to other works on distributed training [START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF][START_REF] Gupta | Distributed learning of deep neural network over multiple agents[END_REF], we assume that the nodes observe data related to the same event. The proposed INL algorithm provides a loss function derived through the theoretical analysis of the best trade-off the model can achieve between the accuracy of the predictions, under logarithmic loss, and the amount of the information passed between the nodes of the communication network.

We also presented a novel scheduling and power allocation algorithms for INL for the star-like topology. The algorithms combine the central nodes' knowledge of the relationships between the data it receives from the edge nodes with the edge nodes' knowledge of the usefulness of the observed data. We showcase, experimentally, the improvement our proposed scheduling and power allocation algorithms provide by combining these two types of knowledge.

Perspectives

Through this thesis, we considered the case in which the compression of the sources is designed separately from the transmission of the data over the channel, the socalled separation of source and channel coding. For the distributed case or the remote inference case, one can improve upon these results by designing the source and channel coding jointly, refereed to as joint source-channel coding (JSCC). Several works are already developing JSCC techniques using NNs for the remote inference case [START_REF] Bourtsoulatze | Deep joint source-channel coding for wireless image transmission[END_REF][START_REF] Farsad | Deep learning for joint source-channel coding of text[END_REF][START_REF] Choi | Neural joint source-channel coding[END_REF],

however, extending this to the distributed cases provides additional challenges. One possible issue is that one might not be able to use gradient descent methods in this case as the channel might not be differentiable with respect to the latent representation transmitted by each node. [START_REF] Zou | Object detection in 20 years: A survey[END_REF], le traitement d'image [START_REF] Pluim | Mutual-information-based registration of medical images: a survey[END_REF], la robotique [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF], et le traitement du langage naturel [START_REF] Vinyals | A neural conversational model[END_REF]. Nous considérons également le cas o ù la performance du modèle est mesurée en termes de perte logarithmique. La perte logarithmique, ou perte logarithmique en abrégé, est une mesure de distorsion naturelle dans les contextes dans lesquels les reconstructions n'ont pas à être déterministes. C'est-à-dire que le décodeur fournit une évaluation de la probabilité de chaque estimation possible, au lieu de choisir une estimation. La mesure est largement utilisée dans divers contextes, y compris le clustering [START_REF] Gur | Variational information bottleneck for unsupervised clustering: Deep gaussian mixture embedding[END_REF] et la classification [START_REF] Alemi | Deep variational information bottleneck[END_REF], la reconnaissance de formes [START_REF] Shkel | Universal lossy compression under logarithmic loss[END_REF], l'apprentissage et la prédiction [START_REF] Cesa-Bianchi | Prediction, learning, and games[END_REF], le traitement d'image [START_REF] Andre | Entropy-based distortion measure for image coding[END_REF], la confidentialité [START_REF] Tripathy | Privacy-preserving adversarial networks[END_REF], et autres. Nous Toutes ces étapes ont ensuite été rapportées et détaillées dans la version intermédiaire [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]. Nous nous différencions de ces travaux en décrivant comment les noeuds locaux peuvent calculer leurs évaluations locales, ainsi qu'en fournissant un moyen approprié pour le noeud de décision de combiner les évaluations locales rec ¸ues avec les valeurs d'activation pour obtenir sa propre évaluation globale. Nous proposons également un algorithme d'ordonnancement amélioré, ainsi qu'un algorithme d'allocation de puissance, qui n'a été présenté ni dans [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF][START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF].

Dans cette thèse, nous proposons de calculer les évaluations locales en utilisant l'information mutuelle (IM) entre les données et la variable cible. Comme le montre [START_REF] Aguerri | Distributed variational representation learning[END_REF],

il existe un lien étroit entre la perte logarithmique moyenne, que nous utilisons dans notre cadre, et le MI. Alors qu'en général MI ne peut pas être calculé analytiquement, on peut l'estimer à l'aide d'échantillons via des estimateurs basés sur NN tels que MINE [START_REF] Ishmael Belghazi | MINE: Mutual Information Neural Estimation[END_REF]. Notre évaluation globale proposée, l'indicateur de pertinence globale (GRI), prend en compte la corrélation entre les représentations compressées que le centre de fusion rec ¸oit, tout en tenant compte des évaluations locales des données observées.

Plus précisément, l'indicateur de pertinence globale est formé en combinant

• les bilans locaux, les MI entre les données et la variable cible, obtenus par chaque noeud,

• évaluations de l'utilité relative des données rec ¸ues de chacun des noeuds.

Le centre de fusion mesure l'utilité relative des données rec ¸ues de chaque noeud à l'aide des informations mutuelles conditionnelles (CMI Mots-clés -apprentissage distribué, théorie de l'information, classification, Information Bottelneck, perte logarithmique, affectation des ressources.

1. 1

 1 Example problem of tracking a vehicle. . . . . . . . . . . . . . . . . . . . 2.1 Relationship between mutual information and entropy. Figure redrawn from [1]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Point-to-point communication model. . . . . . . . . . . . . . . . . . . . . 2.3 Separation of source and channel coding for point-to-point communication model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Remote source coding model. . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 The remote inference problem. Figure adapted from [2]. . . . . . . . . .

3. 1

 1 Studied distributed inference problem. . . . . . . . . . . . . . . . . . . . . 3.2 In-network learning for the network model of Figure 3.1. . . . . . . . . . 3.3 Illustration of the forward and backward passes for an example innetwork learning with J = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Network architecture. Conv stands for a convolutional layer, Fc stands for a fully connected layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Comparison of INL, FL and SL -Experiment 1. . . . . . . . . . . . . . . . viii 3.6 Used NN architecture for FL in Experiment 2. . . . . . . . . . . . . . . . . 3.7 Comparison of INL, FL and SL -Experiment 2. . . . . . . . . . . . . . . . 3.8 CEO problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Studied network inference model. . . . . . . . . . . . . . . . . . . . . . . 4.2 In-network learning and inference using neural networks. . . . . . . . . 4.3 Block diagram of the separate compression-transmission-estimation scheme of Theorem 4.2.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 An example in-network learning with inference fusion and propagation 4.5 Forward and backward passes for the inference problem of Figure 4.4. . 5.1 Wireless sensor networks problem setting. . . . . . . . . . . . . . . . . . . 5.2 In-network learning problem with varying communication channels and data distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Diagram describing the separation between the scheduling and inference phases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 The scheduling and inference phases of our proposed resource allocation solution. Grayed out components represent inactive components. . . . . 5.5 Diagram showcasing experiment setup. . . . . . . . . . . . . . . . . . . . 5.6 Diagram showcasing channel model. . . . . . . . . . . . . . . . . . . . . . 5.7 The accuracy of the system under different scheduling strategies with equal power allocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 The accuracy of the system under different power allocation strategies. . List of Tables 3.1 Bandwidth requirements of INL, FL and SL. . . . . . . . . . . . . . . . . 5.1 NN layers for the encoders and decoder at the local nodes and FC, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 The noise of the view, the channel gain, and the local assessment for each of the nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 The global relevance indicator values obtained for each of the nodes during the scheduling phase. The notation AS signifies nodes that have already been selected, and for which the global relevance indicator values were not computed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x Consider the example problem of identifying or tracking a vehicle in a large urban area. To this end, sensors are placed throughout the area to collect different types of information, such as sound, image/video, proximity, etc. The sensors need to work together to jointly predict where the vehicle is or where it is going. An example of such a problem is illustrated in Figure 1.1.

Figure 1 . 1 :

 11 Figure 1.1: Example problem of tracking a vehicle.

Definition 2 . 1 . 3 .Definition 2 . 1 . 4 .I

 213214 For (X, Y) distributed according to P X,Y , the conditional entropy of X given Y is defined asH(X|Y) =y xp(x, y) log p(x|y).(2.4)Closely related to entropy and conditional entropy is the concept of mutual information between two random variables. Mutual information measures the dependency between two random variables. Informally, mutual information I(X; Y) can be thought of as the amount of information that is common to both variables X and Y. Let (X, Y) ∼ P X,Y be two discrete variables with marginal distributions p(x) = y p(x, y) and p(y) = x p(x, y). The mutual information between X and Y is defined as

Figure 2 . 1 :

 21 Figure 2.1: Relationship between mutual information and entropy. Figure redrawn from [1].

Figure 2 . 2 :

 22 Figure 2.2: Point-to-point communication model.

Figure 2 . 3 :

 23 Figure 2.3: Separation of source and channel coding for point-to-point communication model.

Figure 2 . 4 :

 24 Figure 2.4: Remote source coding model.

Figure 2 . 5 :

 25 Figure 2.5: The remote inference problem. Figure adapted from [2].

Figure 2

 2 Figure 2.6.

Figure 2 . 6 :

 26 Figure 2.6: The trade-off between the relevance and complexity in the IB problem. The solution to the IB problem of (2.26) results in a relevance-complexity pair on the bound of the feasible region (grey area). Figure taken from [3].

. 28 )

 28 We observe that I(Y; U) = H(Y) -H(Y|U) and that H(Y) is fixed by the problem setting, as such maximising the cost function(2.26) over P is equivalent to maximising LIB s (P) := -H(Y|U) -sI(U; X).

. 31 )

 31 Using Lemma 2.3.1 the optimisation of (2.26) can be written in terms of the varia-

Figure 2 . 7 .Figure 2 . 7 :

 2727 Figure 2.7:The diagram shows the variation information bottleneck when we parameterized the distributions using NNs. Given a sample x i the encoding NN outputs the parameters of a distribution, in this case a Gaussian. Samples are then drawn from this distribution and an estimate of y i is obtained by the NN decoding for each sample.

Figure 3 . 1 :

 31 Figure 3.1: Studied distributed inference problem.

Figure 3 . 2 :(

 32 Figure 3.2: In-network learning for the network model of Figure 3.1.

Figure 3 . 3 :

 33 Figure 3.3: Illustration of the forward and backward passes for an example in-network learning with J = 2.

  there are five client nodes. In this setup, we create five sets of noisy versions of the images of CIFAR-10. To this end, the CIFAR images are first normalised and then corrupted by additive Gaussian noise with standard deviation set respectively to 0.4, 1, 2, 3, 4. For our INL each of the five input NNs is trained on a different noisy version of the same image. Each NN uses a variation of the VGG network of [54], with the categorical cross-entropy as the loss function, L2 regularisation, and Dropout and BatchNormalization layers. Node (J + 1) uses two dense layers. The architecture is shown in Figure 3.4. In the experiments, all five (noisy) versions of every CIFAR-10 image are processed simultaneously, each by a different NN at a distinct node, through a series of convolutional layers. The outputs are then concatenated and then passed through a series of dense layers at node (J + 1).

Figure 3 . 4 :

 34 Figure 3.4: Network architecture. Conv stands for a convolutional layer, Fc stands for a fully connected layer.

3 . 4 ,

 34 except the part at Node (J +1)); and node (J +1) is equipped with fully connected layers at Node (J +1) in Figure3.4. Here, the processing during training is such that each input NN concatenates vertically the outputs of all convolution layers and then passes that to node (J + 1), which then propagates back the error vector. After one epoch at one NN, the learnt weights are passed to the next client, which performs the same operations on its part of the dataset. Figure 3.5a depicts the evolution of the (a) Accuracy vs. # of epochs. (b) Accuracy vs. bandwidth cost.

Figure 3 . 5 : 3 . 3 . 3 . 2 2 In Experiment 1 ,

 35333221 Figure 3.5: Comparison of INL, FL and SL -Experiment 1.

Figure 3 . 6 :

 36 Figure 3.6: Used NN architecture for FL in Experiment 2.

  (a) Accuracy vs. # of epochs. (b) Accuracy vs. bandwidth cost.

Figure 3 . 7 :

 37 Figure 3.7: Comparison of INL, FL and SL -Experiment 2.

Figure 3 . 8 :

 38 Figure 3.8: CEO problem.

. 10 ) 3 . 4 . 1 .

 10341 Definition A rate distortion code (of block length n) for the J-encoder model described consists of the encoding functions

25 )

 25 It was show in[START_REF] Aguerri | Distributed variational representation learning[END_REF] that the optimisation of (3.23) can be written in terms of the variational cost function as max

Figure 4 .

 4 1 shows an example of such a network.

Figure 4 . 1 :

 41 Figure 4.1: Studied network inference model.

. 4 )Ŷ

 4 = P(Y); and we measure the discrepancies between the true values of Y ∈ Y and their estimated fits in terms of average logarithmic loss, i.e., for (y, P) ∈ Y × P(Y)

(4. 7 ). 8 ) 4 . 2 . 1 .

 78421 Similarly, for k ∈ [1 : N]/J we have Size of first layer of NN (k)= i : (i,k) ∈ E (Size of last layer of NN (i)). (4Remark Conditions (4.7) and (4.8) were imposed only for the sake of ease of implementation of the training algorithm; the techniques present in this paper, including optimal trade-offs between relevance and complexity for the given topology, the associated loss function, the variational lower bound, how to parameterize it using NNs and so on, do not require (4.7)and (4.8) to hold. (a) Training phase. (b) Inference phase.

Figure 4 . 2 :. 9 ) 4 . 2 . 1 .

 429421 Figure 4.2: In-network learning and inference using neural networks.

  j∈S R j ≥ I(U S ; X S |U S c , Q), for all S ⊆ J (4.11) j∈S∩J R j ≤ C(S) for all S ⊆ [1 : N -1]

15. 5 ]

 5 . The decision node N first recovers the representation codewords (u 1 , . . . , u J ); and, then, produces an estimate of the label y. The scheme is illustrated in Figure 4.3. □ (a) Compression using Berger-Tung coding. (b) Transmission of the bin indices using linear coding.

Figure 4 . 3 :

 43 Figure 4.3: Block diagram of the separate compression-transmission-estimation scheme of Theorem 4.2.1.

Figure 4 . 4 .

 44 Figure 4.4. While the example is simple, it showcases the important aspect of any such topology, the fusion of the data at an intermediary nodes, i.e., a hop.

Figure 4 . 4 :

 44 Figure 4.4: An example in-network learning with inference fusion and propagation

Lemma 4 . 3 . 2 .

 432 For L v-low s defined as in (4.21), L low s as (4.18) and s ≥ 0 max Proof. See Section 4.4.4. □

  by Zaidi, Scaman and Escamilla to the five-node network presented. During the forward pass, every node j ∈ {1, 2, 3} processes mini-batches of size b j of its training data set x j . Nodes 2 and 3 send their vectors formed of the activation values of the last layer of their NNs to node 4, where due to (4.7) the vectors are concatenated vertically at the input layer of NN 4. The forward pass continues on the NN at node 4 until its last layer. Next, nodes 1 and 4 send the activation values of their last layers to node 5. Again, as the sizes of the last layers of the NNs of nodes 1 and 4 satisfy (4.8) the sent activation vectors are concatenated vertically at the input layer of NN 5; and the forward pass continues

Figure 4 . 5 :

 45 Figure 4.5: Forward and backward passes for the inference problem of Figure 4.4.

  to the decision node N over the graph network G = (E, N, C), they are encoded as if they were independent-messages using the linear network coding scheme of [25, Theorem 15.5]; and then transmitted over the network. The transmission of the multimessage (M 1 , . . . , M J ) ∈ [1 : 2 nR 1 ] × . . . × [1 : 2 nR J ] to the decision node N is without error as long as for all S ⊆ [1 : N -1] we have j∈S∩J R j ≤ C(S) (4.26)

Figure 5 . 2 :

 52 Figure 5.2: In-network learning problem with varying communication channels and data distributions.

  not known by the FC in advance, we introduce a scheduling phase during which the importance of each node can be assessed. As a consequence, the transmission during any given block is divided into two periods. The first period, during which all devices transmit and at the end of which the FC selects the K desired devices by allocating resources to them. In what follows, this is called the scheduling phase. The second period, during which only the selected K devices transmit. The FC then performs inference based only on the data it receives from the K devices. In what follows, this is called the inference phase. The two distinct periods are shown in Figure 5.3.

Figure 5 . 3 :

 53 Figure 5.3: Diagram describing the separation between the scheduling and inference phases.

  (a) Power and bandwidth allocation during scheduling phase. (b) Power and bandwidth allocation during inference phase. P * k and B * k are the power and bandwidth allocated to node k during the scheduling phase.

Figure 5 . 4 :

 54 Figure 5.4: The scheduling and inference phases of our proposed resource allocation solution. Grayed out components represent inactive components.

{r 1 ,

 1 ...,r i-1 } j . For simplicity, in the remainder of the chapter we will use the notation GRI r i to refer to GRI {r 1 ,...,r i-1 } r i

  {r 0 ,...,r i-1 } j using r i = arg max j∈J/{r 0 ,...,r i-1 } GRI {r 1 ,...,r i-1 } j 6:

3 . 1

 31 and is estimated locally by the node, while I(Y; U r k ) is estimated during the calculation of the global relevance indicator, more specifically during the ranking of the first node.

Figure 5 . 6 :

 56 Figure 5.6: Diagram showcasing channel model.

Figure 5 . 7 :

 57 Figure 5.7: The accuracy of the system under different scheduling strategies with equal power allocation.

Figure 5 . 8 :

 58 Figure 5.8: The accuracy of the system under different power allocation strategies.

  concentrons sur la formation distribuée et la résolution de problèmes d'inférence. Nous discutons de la fac ¸on dont plusieurs noeuds, qui observent des données liées au même événement, peuvent travailler ensemble pour former conjointement un modèle qui peut être utilisé de manière distribuée pendant la phase d'inférence grâce à un nouveau cadre appelé apprentissage en réseau (INL). INL pour une topologie en étoile, dans laquelle les noeuds qui observent les données sont directement connectés au noeud de décision, a été introduit et analysé pour la première fois dans[START_REF] Aguerri | Distributed variational representation learning[END_REF][START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF]. Nous présentons ces résultats et discutons en détail la procédure de formation pour un tel réseau. Plus précisément, nous discutons comment, dans le cas o ù chaque noeud est équipé d'un NN, un tel réseau peut être formé en utilisant la rétropropagation. Nous expliquons comment, en imposant la condition selon laquelle le NN au centre de fusion utilise comme entrée un vecteur formé en concaténant les informations rec ¸ues des noeuds périphériques, lors de la rétropropagation, le vecteur d'erreur peut être divisé et transmis du centre de fusion à chaque noeud périphérique correspondant. Cette observation a également été faite dans le document de brevet[START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] par A. Zaidi, K. Scaman et P. Escamilla. Nous nous appuyons sur ces résultats en fournissant une nouvelle comparaison expérimentale d'INL avec d'autres algorithmes d'apprentissage distribué, à savoir ceux de[START_REF] Mcmahan | Communication-efficient learning of deep networks from decentralized data[END_REF][START_REF] Gupta | Secure training of multi-party deep neural network[END_REF].Nous considérons ensuite l'extension de l'INL aux topologies de réseau qui peuvent être modélisées avec un graphe dirigé et acyclique. Cela a été proposé et étudié pour la première fois dans le document de brevet[START_REF] Zaidi | In-network learning: Method and system for learning and inference over a telecommunication network[END_REF] par A. Zaidi, K. Scaman et P. Escamilla.Dans cette thèse, nous dérivons une fonction de perte appropriée pour un tel modèle.L'approche de la dérivation de la fonction de perte est empruntée à[START_REF] Aguerri | Distributed variational representation learning[END_REF], c'est-à-dire que nous supposons d'abord que les distributions conjointes sont connues et procédons à la modélisation du problème en tant que source de réseau en codant une sous mesure de perte logarithmique et en trouvant approprié (réalisable) compromis taux-distorsion, puis nous dérivons une limite interne variationnelle, et enfin nous paramétrons cette limite inférieure/interne à l'aide de réseaux de neurones.Nous dérivons d'abord un compromis débit-distorsion réalisable pour toute topologie de réseau qui peut être modélisée avec un graphe acyclique dirigé. La preuve suit en utilisant un schéma de compression-transmission-estimation séparé dans lequel les observations (x 1 , . . . , x J ) sont d'abord compressées de manière distributive en utilisant le codage Berger-Tung[START_REF] Berger | Multiterminal source encoding with one distortion criterion[END_REF] en représentations (u 1 , . . . , u J ); et, ensuite, les indices bin sont transmis sous forme de messages indépendants sur le réseau G en utilisant le codage de réseau linéaire [25, Section 15.5]. Le noeud de décision N récupère d'abord les mots de code de représentation (u 1 , . . . , u J ); et, ensuite, produit une estimation de l'étiquette y. Nous discutons ensuite de la manière dont le compromis réalisable obtenu peut être utilisé, nous dérivons ensuite une limite interne variationnelle pour un exemple de réseau à cinq noeuds. Enfin, nous paramétrons la borne obtenue à l'aide d'un réseau de neurones. Nous discutons ensuite de la manière dont le réseau peut être formé et utilisé pour l'inférence de manière distribuée. Encore une fois, nous discutons comment en imposant la condition que le NN à chaque noeud utilise comme entrée un vecteur formé en concaténant les informations rec ¸ues des noeuds d'ordre inférieur auxquels il est connecté, pendant la rétropropagation, le vecteur d'erreur peut être divisé et transmis de chaque noeud aux noeuds d'ordre inférieur correspondants. Enfin, nous considérons le problème pratique du déploiement d'INL pour une topologie en étoile dans un environnement sans fil contraint; un exemple serait le déploiement dans les réseaux de capteurs sans fil (WSN) [16]. Les performances des WSN sont considérablement limitées par ses ressources radio disponibles et la manière dont les ressources sont allouées entre les noeuds. L'utilisation de techniques d'allocation de ressources développées pour le problème de transmission de données, c'est-à-dire la maximisation de la quantité de données transmises entre les noeuds [20, 75], pour les WSN est sous-optimale pour les tâches d'inférence. Intuitivement, cela est d û au fait que les techniques de transmission de données ne prennent pas en considération la relation entre les données transmises et la tâche d'inférence du WSN. Des chercheurs ont proposé des techniques d'allocation de ressources prenant en compte la relation entre les données transmises et la tâche d'inférence pour les problèmes d'estimation distribuée[START_REF] Xiao | Decentralized estimation in an inhomogeneous sensing environment[END_REF][START_REF] Luo | An isotropic universal decentralized estimation scheme for a bandwidth constrained ad hoc sensor network[END_REF][START_REF] Luo | Universal decentralized estimation in a bandwidth constrained sensor network[END_REF], et de détection distribuée[START_REF] Chamberland | Wireless sensors in distributed detection applications[END_REF][START_REF] Wimalajeewa | Optimal power scheduling for correlated data fusion in wireless sensor networks via constrained pso[END_REF][START_REF] Chen | Channel-aware distributed detection in wireless sensor networks[END_REF]. Ces deux problèmes supposent un modèle prédéfini de la distribution entre les données observées et la variable cible à déduire. Pour pallier l'absence de modèle prédéfini de la répartition entre les données observées et la variable cible, le brevet[START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF] suggérait déjà l'idée d'avoir:1. chaque appareil mesurant sa propre évaluation locale de la pertinence des données qu'il détient pour la tâche d'inférence en cours et la transmet au noeud de décision, 2. cet appareil envoie également simultanément les valeurs d'activation de sortie au noeud de décision, 3. le noeud de décision combinant toutes les informations rec ¸ues (valeurs d'activation des différents appareils ainsi que leurs évaluations locales individuelles) et formant ensuite sa propre évaluation (globale) de la pertinence de l'observation de chaque appareil, 4. application de l'idée à l'ordonnancement des appareils et à l'allocation de puissance (deux algorithmes d'ordonnancement ont déjà été donnés dans[START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF])

Table 3 .

 3

		Federated	Split	In-network
		learning	learning	learning
	Bandwidth requirement 2NJs	2pq + ηNJ s	2pqs J
	VGG 16 50,000 data points	4427 Gbits 324 Gbits	0.16 Gbits
	ResNet 50 50,000 data points	820 Gbits	441 Gbits	0.16 Gbits
	VGG 16 500,000 data points	4427 Gbits 1046 Gbits	1.6 Gbits
	ResNet 50 500,000 data points	820 Gbits	1164 Gbits	1.6 Gbits

1: Bandwidth requirements of INL, FL and SL.

  . . , P U J |X J ,Q , P Q } for which For each tuple (∆, R sum ) on the boundary of the relevance complexity region RI sum DIB there exists s ≥ 0 such that (∆, R sum ) = (∆ s , R s ) where

	the resulting (∆, R sum ) is on the boundary of the RI sum DIB . The following proposition
	provides such a characterisation.
	Proposition 3.4.1. [10, Proposition 2] ∆ s := 1 1 + s (1 + sK)H(Y) + sR s + max P	L DIB s (P)]	(3.21)

  .18) The following lemma shows that L low s (P, P U 4 |U 2 ,U 3 ) lower bounds L s (P) as given by (4.16).For convenience letP + := {P U 1 |X 1 , P U 2 |X 2 , P U 3 |X 3 , P U 4 |U 2 ,U 3 }.The optimisation of (4.18)generally requires the computation of marginal distributions, which can be costly in practice. Hereafter we derive a variational lower bound on L low

	s	with respect to some
	arbitrary (variational) distributions. Specifically, let
	Lemma 4.3.1. For every s ≥ 0 and joint measure that factorises as (4.17), we have
	L s (P) ≥ L low s (P, P U 4 |U 2 ,U 3 ),	(4.19)

Proof. See Section 4.4.3.

□

  .2)In most practical applications, calculating the MI I(Y; U 1 , . . . , U J ) and CMI I(Y; U j |U K p ) is not possible; in such cases, the values must be estimated using the available data. Let Î(Y; U j |U K p ) denote such an estimate of I(Y; U j |U K p ) and Î(Y; U 1 , . . . , U J ) denote such an estimate of I(Y; U 1 , . . . , U J ). We now define the approximation of the global relevance |U K p ), using the trained NN at the FC. We first split the term into the difference between two conditional entropies

	indicator using
	GRI	K p j =	RI j J i=1 RI i	Î(Y; U j |U K p ) Î(Y; U 1 , . . . , U J )	(5.3)
	Any data-based technique can be used to estimate the MI and CMI terms, including
	those presented in [69-71] and references within. In what follows, we detail a method
	for estimating I(Y; U j

Table 5 .

 5 2 shows the channel gains and RIs for each node.

	Node ID Data Noise h 2 j	RI j
	Node 1	No Noise	1.e-01 0.27
	Node 2	N(0; 0.7)	7.e-02 1.13
	Node 3	N(0; 0.1)	2.e-02 1.11
	Node 4	N(0; 0.6)	6.e-04 1.4
	Node 5	N(0; 0.3)	1.e+00 0.43
	Node 6	N(0; 0.1)	1.e-05 0.44
	Node 7	N(0; 1.5)	3.e-03 0.64
	Node 8	N(0; 0.2)	1.e-06 1.65
	Node 9	N(0; 0.2)	1.e-05 0.43

Table 5 .

 5 2:The noise of the view, the channel gain, and the local assessment for each of the nodes.

Table 5

 5

	.3 the three chosen nodes are nodes 2, 4 and 7. The first
	node to be selected is node 2. The choice intuitively makes sense, node 2 observes a

central part of the image, thus containing a lot of information about the target ariable, i.e., the class. This can be observed from the RI; compared to nodes that observe parts of the image at the periphery, node 2 has a better local assessment, i.e., higher RI, see Table 5.2. Compared to other nodes that observe central parts of the image, node 2 has

Table 5 .

 5 3: The global relevance indicator values obtained for each of the nodes during the scheduling phase. The notation AS signifies nodes that have already been selected, and for which the global relevance indicator values were not computed. a better channel.The second node to be chosen is node 4. Compared to the other nodes, the global relevance indicator of node 4, given that node 2 was selected, is the highest. Node 4 has the highest local assessment compared to the other nodes, all except node 8 which has

  Another possible research avenue is the development of an extension of INL in which nodes exchange quantised values during training and inference. This would allow the deployment of INL in digital communication channels and would significantly reduce communication constraints. The challenge of implementing such a techniqueis the fact that a discreet latent representation would again prevent the use of gradient descent methods. Although there are techniques that allow neural network training even when the latent representation is discreet, see[START_REF] Choi | Neural joint source-channel coding[END_REF][START_REF] Hanna | On distributed quantization for classification[END_REF][START_REF] Van Den Oord | Neural discrete representation learning[END_REF], these techniques are not appropriate for the problem setting considered in INL. The authors of[START_REF] Choi | Neural joint source-channel coding[END_REF] consider only the problem of communication, and not inference, and in[START_REF] Hanna | On distributed quantization for classification[END_REF] the authors have shown that the quantisation of the latent space should take into account the task that the model seeks to solve. The techniques of both[START_REF] Hanna | On distributed quantization for classification[END_REF][START_REF] Van Den Oord | Neural discrete representation learning[END_REF] can be used for developing an INL model with a discreet latent representation; however, they require that the model be trained locally, which is not the case in INL. Prenons l'exemple du problème d'identification ou de suivi d'un véhicule dans une grande zone urbaine. Des capteurs sont placés dans toute la zone pour collecter différents types d'informations, telles que le son, l'image/vidéo, la proximité, etc. Les capteurs doivent fonctionner ensemble pour prédire conjointement o ù se trouve le véhicule ou o ù il va. Les données recueillies par les capteurs sont corrélées. La relation exacte entre les données que nous observons et les données que nous voulons prédire n'est pas connue. Nous supposons que les capteurs collectent des données à partir desquelles nous pouvons former un modèle pour apprendre la relation entre les données observées et les informations que nous voulons prédire. Nous appelons cette période de temps la phase de formation. La collecte des échantillons de données des capteurs vers un emplacement central n'est pas toujours possible. Les capteurs peuvent être déployés dans une zone o ù l'infrastructure de communication est limitée [4]. La question est, compte tenu de certaines mesures de la performance du modèle, quelles données chaque noeud doit-il extraire de ses données observées? La suppression de l'une des données pertinentes ne peut que réduire les performances de prédiction. D'autre part, l'extraction et la transmission de données qui ne sont pas pertinentes pour la tâche à accomplir peuvent mettre à rude épreuve le réseau de communication ou même dégrader les performances du réseau. Ainsi, intuitivement, les noeuds doivent extraire le plus petit nombre de caractéristiques qui donnent collectivement les meilleures performances de prédiction. Décider quelles fonctionnalités extraire doit être fait sans coordination explicite entre les noeuds. Comment cela devrait-il être effectué de manière optimale sans partager de données brutes? En particulier, comment chaque noeud doit-il apprendre à traiter les informations et que doit-il transmettre aux autres noeuds de manière distribuée? Dans cette thèse, nous supposons qu'il existe des dispositifs, que nous désignons comme noeuds périphériques, qui observent des données liées à une variable d'intérêt. Cette variable d'intérêt doit être déduite à un noeud de décision central ou à un centre de fusion. Les noeuds périphériques sont connectés au noeud central soit directement, soit via des noeuds intermédiaires. Nous supposons que la topologie du réseau peut être modélisée par un graphe orienté acyclique. La distribution des données n'est pas connue et seul un jeu de données distribué est disponible; c'est-à-dire que différentes parties des données sont disponibles à différents noeuds. Nous équipons chaque noeud d'un réseau de neurones (NN). Cela est d û au succès des techniques d'apprentissage automatique (ML) dans l'apprentissage, à partir d'échantillons, de relations complexes entre des données observées et une variable cible dans des domaines tels que la vision par ordinateur

	French Summary(Résumé Franc ¸ais)

Un autre problème possible est que des informations sensibles peuvent être contenues dans les données collectées. Le transfert des données pourrait violer les restrictions de confidentialité

[START_REF] Horvitz | Data, privacy, and the greater good[END_REF]

. Pour ces raisons, tant la phase d'apprentissage que le déploiement du modèle, également appelée phase d'inférence, doivent se dérouler de manière distribuée, c'est-à-dire sans échange de données brutes. Ce problème est parfois appelé problème d'apprentissage distribué et d'inférence.

  ). Pour calculer l'utilité des données du noeud j, par rapport au noeud i, le centre de fusion a calculé le MI des représentations latentes obtenues du noeud j et de la variable cible, compte tenu des représentations latentes du noeud i. Le CMI est ensuite mis à l'échelle par le MI entre la variable cible et la représentation latente obtenue à partir de tous les noeuds périphériques. L'algorithme de planification cherche à sélectionner les noeuds avec le GRI le plus élevé. En tant que tel, le FC calcule le GRI de chaque noeud, compte tenu des noeuds déjà sélectionnés, puis sélectionne le noeud avec le GRI le plus élevé. Le processus est répété jusqu'à ce que les noeuds soient planifiés. L'algorithme d'allocation de puissance cherche ensuite à trouver un équilibre entre la maximisation des informations totales que les noeuds sélectionnés transmettent au FC, l'utilité des informations transmises pour déduire la variable cible et la quantité totale d'informations utiles concernant la variable cible que chaque noeud peut transmettre. au FC pendant l'inférence. Pour atteindre ce compromis souhaité, nous introduisons une mesure, notée utilité attendue (EU), qui pour chaque noeud k, combine l'utilité du noeud pour inférer la variable cible sous une allocation de puissance égale, et la quantité d'informations concernant la cible variable qui est contenue dans les données observées mais qui n'a pas pu être transmise, sous allocation de puissance égale. L'utilité du noeud pour déduire la variable cible sous allocation de puissance égale est mesurée par l'évaluation de l'utilité relative du noeud qui a été calculée pour le GRI pendant la phase de programmation. La quantité d'informations concernant la variable cible qui est contenue dans les données observées mais qui n'a pas pu être transmise est mesurée par le CMI entre la variable cible et les données observées compte tenu de la représentation latente transmise, qui est ensuite mise à l'échelle par le MI entre la variable cible et les données observées. La mesure de l'EU est obtenue en prenant le produit entre l'utilité du noeud pour déduire la variable cible sous une allocation de puissance égale, et la quantité d'informations concernant la variable cible qui est contenue dans les données observées mais n'a pas pu être transmise. La puissance est ensuite distribuée proportionnellement à l'EU de chaque noeud.

The function is defined up to a multiplicative constant.

It is interesting to note that the original formulation of the IB problem did not consider the connection between the generalisation gap and the description length of the latent representation. This connection was found later.

The optimality is proved therein under the assumption that for every subset S ⊆ J it holds that X S -Y -X S c . The RHS of (3.6) is achievable for arbitrary distributions, however, regardless of such an assumption.

For simplicity η and σ are assumed here to be identical for all NNs.

We assume all the elements of X j have the same dimension.

The inference problem is a one-shot problem. The result of Theorem 4.2.1 is asymptotic in the size of the training data-sets. One-shot results for this problem can be obtained, e.g., along[START_REF] Li | Strong functional representation lemma and applications to coding theorems[END_REF].

The time sharing random variable is set to a constant for simplicity.

For simplicity η and σ are assumed here to be identical for all NNs.

Ideally the number K itself should be subject to design, but we do not delve into that in this chapter.

Chapter 5

Scheduling and Resource Allocation

In this chapter we consider the problem of resource allocation for the case in which the INL algorithm is deployed in a wireless sensor network (WSN). A solution of such a problem was first presented in the patent document [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF] by Zaidi, and then described in more detail in Zaidi-Krasnowski [START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF]. Before introducing the results of [START_REF] Zaidi | Joint relevance-channel aware device scheduling for in-network learning[END_REF][START_REF] Zaidi | Channel-aware device scheduling and power allocation for distributed inference[END_REF] we quickly describe the problem and highlight the challenges in solving it. We again consider the problem shown in Figure 5.1. The fusion centre (FC) must infer an estimate of a random variable of interest, Y, from the observations X 1 , . . . , X J acquired/measured each at a distinct device/node. In addition, it is assumed that the transmission between the devices and the FC is resource-constrained. That is, the devices are not allowed to (e.g., due to communication or privacy constraints) share their raw data, X j , with the FC. As a consequence, the FC needs to output an estimate Ŷ FC selects a subset of K ≤ J devices 1 to be scheduled for use during inference. The scheduling algorithm is detailed in Section 5.3.3.

Subsequently, the FC allocates the total available power to the K selected nodes. This is shown in Figure 5.4b. Each node is allocated power proportionally to the quality of its communication channel and its assigned global assessment. The power allocation procedure is discussed in Section 5.3.4.

Local Assessment

In this section, we describe how each node can compute its local assessment of the data it observes, the RI j . Other works have shown the usefulness of local assessments for the problem of scheduling or resource allocation. In [START_REF] Chen | Channel-aware distributed detection in wireless sensor networks[END_REF]Example 2], the authors give an example where a distributed quantizor performs best when it is designed based on the knowledge of both the channel and the local distribution. In [START_REF] Zhang | Optimal power allocation for distributed detection over mimo channels in wireless sensor networks[END_REF], the authors present experiments where power allocation performs best when both the channel quality and the local assessment of the data are taken into account.

We argue that one can always find a suitable relevance indicator for a given task.

In [START_REF] Xiao | Decentralized estimation in an inhomogeneous sensing environment[END_REF], the local assessment is calculated using the SNR. For the purpose of scheduling devices for distributed training, in [START_REF] Liu | Data-importance aware user scheduling for communication-efficient edge machine learning[END_REF], the authors introduce two uncertainty measures, as viewed by the model under training, while the authors of [START_REF] Amiri | Update aware device scheduling for federated learning at the wireless edge[END_REF] use the significance of the model updates to assess the importance of each node.

In this chapter, we use the mutual information (MI) between the data and the target variable, I(Y; X j ), to obtain the local assessment at each node. As shown in [START_REF] Aguerri | Distributed variational representation learning[END_REF], there is a strong connection between the average logarithmic loss, which we use in our framework, and the MI. While in general MI cannot be computed analytically, it can be estimated using samples using NN-based estimators such as MINE [START_REF] Ishmael Belghazi | MINE: Mutual Information Neural Estimation[END_REF].

These examples are not exhaustive. Other functions/techniques can be used together with the algorithms we provide in Section 5.3.3 for scheduling devices. The chosen local assessment function must meet three main characteristics. Firstly, the resulting output should be a value that can be sent over a side channel with a much smaller rate than the primary channel. Secondly, data that contain more useful information allocation strategy offers. We consider a variation of the problem of classifying MNIST [START_REF] Deng | The mnist database of handwritten digit images for machine learning research[END_REF] digits. In our setup, there are nine local nodes connected to one FC; each local node observes a noisy segment of the MNIST image. Eight of the nodes observe segments that are nonoverlapping and of dimension 14x7 pixels. One node, node 3, observes a noisy version of the same image observed by node 2. Each segment is corrupted with zero-mean additive Gaussian noise of different variance; Table 5.2 shows the noise distribution for each node. The setup is shown in Figure 5.5. Each node j ∈ [1 : 9] is equipped with a deterministic NN-based encoder, the architecture of which is shown in Table 5.1, which, for a given input x j,i outputs a latent vector u 50 j,i = [u 1 j,i , . . . , u 50 j,i ]. Since the following theory applies to all samples i ∈ [1 : n] we will drop the sample subscript for simplicity. For reasons that will become clearer, in what follows, we normalise the vector u 50 j during both inference and training. The FC is similarly equipped with an NN-based decoder whose architecture is shown in Table 5.1. We train the models using categorical cross-entropy with a batch size of 512 for 100 epochs.

Experimental Setup

Each node j needs to transmit each element of the latent vector u 50 j over a constrained communication channel of capacity C j , given by equation (5.1). We consider separate source-channel coding; Figure 5.6 shows a diagram of the communication channel.

Due to the communication constraints, the source encoder needs to compress the data (5.12)

For ease of calculation, we model the latent variable as i.i.d samples from a Gaussian distribution of zero mean and unit variance. To ensure that the latent vector has zero mean and unit variance, we normalise u 50 j during both the training and inference procedure. To bypass the issue of developing and fixing the codebooks prior to training, we approximate the codewords by adding noise to each instance of the latent variable.

We make this approximation, as in this setup we are interested only in finding a relationship between the distortion caused by the compression of the latent variable and the channel capacity. More specifically, we compute the vector received by the FC from node j ∈ J, which we denote by û50 j , as ût j = u t jz t j ∀ t ∈ [0 : 50] with (5.13) z t j ∼ N u t j 1 + h 2 j P j ; 1 1 + h 2 j P j -1 (1 + h 2 j P j ) 2 .

(5.14)

This approximation is derived based on the lossy source coding of Gaussian sources [START_REF] Gamal | Network Information Theory[END_REF]. More specifically, we use the fact that to compress a source U ∼ N(0, σ 2 U ) into Û,