Yassine Ouali
email: yassine.ouali@centralesupelec.fr

Céline Hudelot
email: celine.hudelot@centralesupelec.fr

Myriam Tami
email: myriam.tami@centralesupelec.fr

An Overview of Deep Semi-Supervised Learning

Keywords: semi-supervised learning, deep learning, neural networks, consistency training, entropy minimization, proxy labeling, generative models, graph neural networks

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

In recent years, semi-supervised learning (SSL) has emerged as an exciting new research direction in deep learning. Such methods deal with the situation where few labeled training examples are available together with a significant number of unlabeled samples. In such a setting, SSL methods are more applicable to realworld applications where the unlabeled data are readily available and easy to acquire, while labeled instances are often hard, expensive, and time-consuming to collect. SSL is capable of building better classifiers that compensate for the lack of labeled training data. However, in order to avoid a lousy matching of the problem structure with the model assumption, which can lead to a degradation in classification performance [START_REF] Xiaojin | Semi-supervised learning literature survey[END_REF], SSL is only effective under certain assumptions, such as assuming that the decision boundary should avoid regions with high density, facilitating the extraction of additional information from the unlabeled instances to regularize training. In this paper, we will start by an introduction to SSL with its main assumptions and methods, followed by a summarization of the dominant semi-supervised approaches in deep learning. For a detailed and comprehensive review of the field, Semi-Supervised Learning Book [START_REF] Chapelle | Semi-supervised learning[END_REF] is a good resource. "Semi-supervised learning (SSL) is halfway between supervised and unsupervised learning. In addition to unlabeled data, the algorithm is provided with some supervision information -but not necessarily for all examples. Often, this information will be the targets associated with some of the examples. In this case, the data set X = (x i ); i ∈ [n] can be divided into two parts: the points X l := (x 1 , . . . , x l ), for which labels Y l := (y 1 , . . . , y l ) are provided, and the points X u := (x l+1 , . . . , x l+u ), the labels of which are not known." -Chapelle et al. [START_REF] Chapelle | Semi-supervised learning[END_REF].

Semi-supervised learning

As stated in the definition above, in SSL, we are provided with a dataset containing both labeled and unlabeled examples. The portion of labeled examples is usually quite small compared to the unlabeled example (e.g., 1 to 10% of the total number of examples). So with a dataset D containing a labeled subset D l and an unlabeled subset D u , the objective, or rather hope, is to leverage the unlabeled examples to train a better performing model than what can be obtained using only the labeled portion. And hopefully, get closer to the desired optimal performance, in which all of the dataset D is labeled.

More formally, the goal of SSL is to leverage the unlabeled data D u to produce a prediction function f θ with trainable parameters θ, that is more accurate than what would have been obtained by only using the labeled data D l . For instance, D u might provide us with additional information about the structure of the data distribution p(x) to better estimate the decision boundary between the different classes. For example, as shown in fig. 1, where the data points with distinct labels are separated with a low-density region, leveraging unlabeled data with a SSL approach can provide us with additional information about the shape of the decision boundary between two classes, and reduce the ambiguity present in the supervised case.

SSL first appeared in the form of self-training [START_REF] Chapelle | Semi-supervised learning[END_REF], which is also known as self-labeling or self-teaching. A model is first trained on labeled data. Then, iteratively, a portion of the unlabeled data is annotated using the trained model and added to the training set for the next training iteration. SSL took off in the 1970s after its success with iterative algorithms such as the expectation-maximization algorithm [START_REF] Todd K Moon | The expectation-maximization algorithm[END_REF], in which the labeled and unlabeled data are jointly used to maximize the likelihood of the model.

SSL Methods

There have been many SSL methods and approaches that have been introduced over the years. These algorithms can be broadly divided into the following categories:

• Consistency Regularization (a.k.a Consistency Training). Based on the assumption that if a realistic perturbation was applied to the unlabeled data points, the prediction should not change significantly. The model can then be trained to have a consistent prediction on a given unlabeled example and its perturbed version.

• Proxy-label Methods. Such methods leverage a trained model on the labeled set to produce additional training examples by labeling instances of the unlabeled set based on some heuristics. These approaches can also be referred to as bootstrapping [START_REF] Biemann | Unsupervised and knowledge-free natural language processing in the structure discovery paradigm[END_REF] algorithms. We follow Ruder et al. [START_REF] Ruder | Strong baselines for neural semi-supervised learning under domain shift[END_REF] and refer to them as proxy-label methods. Some examples of such methods are Self-training, Co-training and Multi-View Learning.

• Generative Models. Similar to the supervised setting, where the learned features on one task can be transferred to other downstream tasks. Generative models that are able to generate images from the data distribution p(x) must learn transferable features to a supervised task p(y|x) for a given task with targets y.

• Graph-Based Methods. The labeled and unlabeled data points can be considered as nodes of a graph, and the objective is to propagate the labels from the labeled nodes to the unlabeled ones by utilizing the similarity of two nodes x i and x j , which is reflected by how strong the edge e ij between the two nodes.

In addition to these main categories, there is also some SSL work on entropy minimization, where we force the model to make confident predictions by minimizing the entropy of the predictions. Consistency training can also be considered a proxy-label method, with a subtle difference, instead of considering the predictions as ground-truths and compute the cross-entropy loss, we enforce consistency of predictions by minimizing a given distance between the outputs. SSL methods can also be categorized based on two dominant learning paradigms, transductive learning and inductive learning. Transductive learning aims to apply the trained classifier on the unlabeled instances observed at training time; in this case, it does not generalize to unobserved instances. This type of algorithm is mainly used on graphs, such as random walks for node embedding [START_REF] Perozzi | Deepwalk: Online learning of social representations[END_REF][START_REF] Grover | node2vec: Scalable feature learning for networks[END_REF], where the objective is to label the unlabeled nodes of the graph that are present at training time. The more popular paradigm, inductive learning, aims to learn a classifier capable of generalizing to unobserved instances at test time.

Main Assumptions in SSL

The first question we need to answer is under what assumptions can we apply SSL algorithms? SSL algorithms only work under some assumptions about the structure of the data need to hold. Without such assumptions, it would not be possible to generalize from a finite training set to a set of possibly infinitely many unseen test cases. The main assumptions in SSL are:

• The Smoothness Assumption. If two points x 1 , x 2 reside in a high-density region are close, then so should be their corresponding outputs y 1 , y 2 [START_REF] Chapelle | Semi-supervised learning[END_REF]. Meaning that if two inputs are of the same class and belong to the same cluster, which is a high-density region of the input space, then their corresponding outputs need to be close. The inverse also holds true; if the two points are separated by a low-density region, the outputs must be distant from each other. This assumption can be quite helpful in a classification task, but not so much for regression.

• The Cluster Assumption. If points are in the same cluster, they are likely to be of the same class [START_REF] Chapelle | Semi-supervised learning[END_REF]. In this particular case of the smoothness assumption, we suppose that input data points form clusters, and each cluster corresponds to one of the output classes. The cluster assumption can also be seen as the low-density separation assumption: The decision boundary should lie in the low-density regions. The relation between the two assumptions is easy to see, if a given decision boundary lies in a high-density region, it will likely cut a cluster into two different classes, resulting in samples from different classes belonging to the same cluster, which is a violation of the cluster assumption. In this case, we can restrict our model to have consistent predictions on the unlabeled data over some small perturbations pushing its decision boundary to low-density regions.

• The Manifold Assumption. The (high-dimensional) data lie (roughly) on a low-dimensional manifold [START_REF] Chapelle | Semi-supervised learning[END_REF]. In high dimensional spaces, where the volume grows exponentially with the number of dimensions, it can be quite hard to estimate the true data distribution for generative tasks. For discriminative tasks, the distances are similar regardless of the class type, making classification quite challenging. However, if our input data lies on some lower-dimensional manifold, we can try to find a low dimensional representation using the unlabeled data and then use the labeled data to solve the simplified task.

Related Problems

Active Learning In active learning [START_REF] Settles | Active learning literature survey[END_REF][START_REF] Hanneke | Theoretical foundations of active learning[END_REF], the learning algorithm is provided with a large pool of unlabeled data points, with the ability to request the labeling of any given examples from the unlabeled set in an interactive manner. As opposed to classical passive learning, in which the examples to be labeled are chosen randomly from the unlabeled pool, active learning aims to carefully choose the examples to be labeled to achieve a higher accuracy while using as few requests as possible, thereby minimizing the cost of obtaining labeled data. This is of particular interest in problems where data may be abundant, but labels are scarce or expensive to obtain.

Although it is not possible to obtain a universally good active learning strategy [START_REF] Dasgupta | Analysis of a greedy active learning strategy[END_REF], there exist many heuristics [START_REF] Settles | Active learning literature survey[END_REF], which have been proven to be effective in practice. The two widely used selection criteria are informativeness and representativeness [START_REF] Huang | Active learning by querying informative and representative examples[END_REF][START_REF] Zhou | A brief introduction to weakly supervised learning[END_REF]. Informativeness measures how well an unlabeled instance helps reduce the uncertainty of a statistical model, while representativeness measures how well an instance helps represent the structure of input patterns.

Active learning and SSL are naturally related, since both aim to use a limited amount of data to improve a learner. Several works considered combining SSL and AL in different tasks. [START_REF] Drugman | Active and semi-supervised learning in asr: Benefits on the acoustic and language models[END_REF] demonstrates a significant error reduction with limited labeled data for speech understanding, [START_REF] Kyu Rhee | Active and semisupervised learning for object detection with imperfect data[END_REF] proposes an active semi-supervised learning system for pedestrian detection, [START_REF] Zhu | Combining active learning and semi-supervised learning using gaussian fields and harmonic functions[END_REF] combines AL and SSL using Gaussian fields applied to synthetic datasets, and [START_REF] Gao | Consistency-based semi-supervised active learning: Towards minimizing labeling cost[END_REF] exploits both labeled and unlabeled data using SSL to distill information from unlabeled data that improves representation learning and sample selection.

Transfer Learning and Domain Adaptation Transfer learning [START_REF] Sinno | A survey on transfer learning[END_REF][START_REF] Weiss | A survey of transfer learning[END_REF] is used to improve a learner on one domain, called the target domain, by transferring the knowledge learned from a related domain, referred to as the source domain. For instance, we may wish to train the model on a synthetic, cheap-togenerate data, with the goal of using it on real data. In this case, the source domain used to train the model is related but different from the target domain used to test the model. When the source and target differ but are related, then transfer learning can be applied to obtain higher accuracy on the target data.

One popular type of transfer learning is domain adaptation [START_REF] Quionero-Candela | Dataset shift in machine learning[END_REF][START_REF] Vishal | Visual domain adaptation: A survey of recent advances[END_REF][START_REF] Wilson | A survey of unsupervised deep domain adaptation[END_REF]. Domain adaptation is a type of transductive transfer learning, where the target task remains the same as the source, but the domain differs. The objective of domain adaptation is to train a learner capable of generalizing across different domains of different distributions in which the labeled data are available for the source domain. As for the target domain, we refer to the case where no labeled data is available on target as unsupervised domain adaptation, while semi-supervised and supervised domain adaptation refers to situations where we have a limited or a fully labeled target domain receptively [START_REF] Beijbom | Domain adaptations for computer vision applications[END_REF]. SSL and unsupervised domain adaptation are closely related; in both cases, we are provided with labeled and unlabeled data, with the objective of learning a function capable of generalizing to the unlabeled data and unseen examples. However, in SSL, both the labeled and unlabeled sets come from the same distribution, while in unsupervised domain adaptation, the target and source distributions differ. Methods in both subjects can be leveraged interchangeably. In SSL, [START_REF] Mayer | Adversarial feature distribution alignment for semi-supervised learning[END_REF] proposed to use adversarial distribution alignment [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF] for semi-supervised image classification using only a small amount of labeled samples. As for unsupervised domain adaptation, semi-supervised methods, such as consistency regularization [START_REF] Shu | A dirt-t approach to unsupervised domain adaptation[END_REF][START_REF] Lee | Drop to adapt: Learning discriminative features for unsupervised domain adaptation[END_REF][START_REF] French | Self-ensembling for visual domain adaptation[END_REF], co-regularization [START_REF] Kumar | Co-regularized alignment for unsupervised domain adaptation[END_REF] or proxy labeling [START_REF] Saito | Asymmetric tri-training for unsupervised domain adaptation[END_REF][START_REF] Ruder | Strong baselines for neural semi-supervised learning under domain shift[END_REF] demonstrated their effectiveness in domain adaptation.

Weakly-Supervised Learning To overcome the need for large hand-labeled and expensive training sets, most sizeable deep learning systems use some form of weak supervision: lower-quality, but larger-scale training sets constructed via strategies such as using cheap annotators [START_REF] Ratner | Weak supervision: A new programming paradigm for machine learning[END_REF]. In weakly-supervised learning, the objective is the same as in supervised learning, however, instead of a ground-truth labeled training set, we are provided with one or more weakly annotated examples, that could come from crowd workers, be the output of heuristic rules, the result of distant supervision [START_REF] Mintz | Distant supervision for relation extraction without labeled data[END_REF], or the output of other classifiers. For example, in weakly-supervised semantic segmentation, pixel-level labels, which are harder and more expensive to acquire, are substituted for inexact annotations, e.g., image labels [START_REF] Wang | Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features[END_REF][START_REF] Zhou | Learning deep features for discriminative localization[END_REF][START_REF] Wei | Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation[END_REF][START_REF] Li | Tell me where to look: Guided attention inference network[END_REF][START_REF] Lee | Ficklenet: Weakly and semi-supervised semantic image segmentation\\using stochastic inference[END_REF], points [START_REF] Bearman | What's the point: Semantic segmentation with point supervision[END_REF], scribbles [START_REF] Lin | Scribblesup: Scribble-supervised convolutional networks for semantic segmentation[END_REF] and bounding boxes [START_REF] Song | Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation[END_REF][START_REF] Dai | Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation[END_REF]. In such a scenario, SSL approaches can be used to enhance the performance further if a limited number of strongly labeled examples are available while still taking advantage of the weakly labeled examples.

Learning with Noisy Labels Learning from noisy labels [START_REF] Frénay | Classification in the presence of label noise: a survey[END_REF][START_REF] García | Data preprocessing in data mining[END_REF] can be challenging given the negative impact label noise can have on the performance of deep learning methods if the noise is significant. To overcome this, most existing methods for training deep neural networks with noisy labels seek to correct the loss function. One type of correction consists of treating all the examples as equal and relabeling the noisy examples, where proxy labels methods can be used for the relabeling procedure [START_REF] Yi | Probabilistic end-to-end noise correction for learning with noisy labels[END_REF][START_REF] Ma | Dimensionality-driven learning with noisy labels[END_REF][START_REF] Reed | Training deep neural networks on noisy labels with bootstrapping[END_REF]. Another type of correction applies a reweighing to the training examples to distinguish between the clean and noisy samples [START_REF] Cooke | Robust automatic speech recognition with missing and unreliable acoustic data[END_REF][START_REF] Sunil Thulasidasan | Combating label noise in deep learning using abstention[END_REF]. Other works [START_REF] Ding | A semi-supervised two-stage approach to learning from noisy labels[END_REF][START_REF] Hendrycks | Augmix: A simple data processing method to improve robustness and uncertainty[END_REF][START_REF] Kong | Recycling: Semisupervised learning with noisy labels in deep neural networks[END_REF][START_REF] Li | Dividemix: Learning with noisy labels as semi-supervised learning[END_REF] have shown that SSL can be useful in learning from noisy labels, where the noisy labels are discarded, and the noisy examples are considered as unlabeled data and used to regularize training using SSL methods.

Evaluating SSL Approaches

The conventional experimental procedure used to evaluate SSL methods consists of choosing a dataset (e.g., , SVHN [START_REF] Netzer | Reading digits in natural images with unsupervised feature learning[END_REF], ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], IMDb [START_REF] Andrew L Maas | Learning word vectors for sentiment analysis[END_REF], Yelp review [START_REF] Zhang | Character-level convolutional networks for text classification[END_REF]) commonly used for supervised learning, a large portion of the labels are then ignored, resulting in a small labeled set D l and a larger unlabeled D u . A deep learning model is trained with a given SSL approach, and the results are reported on the original test set over various and standardized portions of labeled examples. In order to make this procedure applicable to real-world settings, Oliver et al. [START_REF] Oliver | Realistic evaluation of deep semi-supervised learning algorithms[END_REF] proposed the following ways to improve this experimental methodology:

• A Shared Implementation. For a realistic comparison of different SSL methods, they must share the same underlying architectures and other implementation details (e.g., hyperparameters, parameter initialization, data augmentation, regularization, etc.).

• High-Quality Supervised Baseline. The main objective of SSL is to obtain better performance than what can be obtained in a supervised manner. This is why it is essential to provide a strong baseline consisting of training the same model on the labeled set D l in a supervised way, with modified hyperparameters to report the best-case performance of the fully-supervised model.

• Comparison to Transfer Learning. Another robust baseline to compare SSL methods to can be obtained by training the model on large labeled datasets, and then fine-tune it on the small labeled set D l .

• Considering Class Distribution Mismatch. The possible distribution mismatch between the labeled and unlabeled examples can be ignored when doing evaluation since both sets come from the same dataset. Still, such a mismatch is prevalent in real-world applications, where the unlabeled data can have different class distributions compared to the labeled data. The effect of this discrepancy needs to be addressed for better real-world adoption of SSL.

• Varying the Amount of Labeled and Unlabeled Data. A common practice in SSL is varying the number of labeled examples, but also varying the size D u in a systematic way to simulate realistic scenarios, such as training on a relatively small unlabeled set, can provide additional insights into the effectiveness of SSL approaches.

• Realistically Small Validation Sets. In many cases where a fully annotated dataset if used for evaluation, we might end-up with a validation set that is significantly larger than the labeled set D l used for training, in such a setting, extensive hyperparameter tuning might result in an overfitting to the validation set. In contrast, small validation sets constrain the ability to select models [START_REF] Chapelle | Semi-supervised learning[END_REF][START_REF] Forster | Neural simpletrons: Learning in the limit of few labels with directed generative networks[END_REF], resulting in a more realistic assessment of the performance of SSL methods.

Consistency Regularization

A recent line of works in deep semi-supervised learning utilizes the unlabeled data to enforce the trained model to be in line with the cluster assumption, i.e., , the learned decision boundary must lie in low-density regions. These methods are based on a simple concept that, if a realistic perturbation was to be applied to an unlabeled example, the prediction should not change significantly, given that under the cluster assumption, data points with distinct labels are separated with low-density regions, so the likelihood of one example to switch classes after a perturbation is small (e.g., fig. 1).

More formally, with consistency regularization, we are favoring functions f θ that give consistent predictions for similar data points. So rather than minimizing the classification cost at the zero-dimensional data points of the inputs space, the regularized model minimizes the cost on a manifold around each data point, pushing the decision boundaries away from the unlabeled data points and smoothing the manifold on which the data resides [START_REF] Xiaojin | Semi-supervised learning literature survey[END_REF]. Concretely, given an unlabeled data point x ∈ D u and its perturbed version xu , the objective is to minimize the distance between the two outputs d(f θ (x), f θ (x)). The popular distance measures d are mean squared error (MSE), Kullback-Leiber divergence (KL) and Jensen-Shannon divergence (JS). For two outputs f θ (x) and f θ (x) in the form of a probability distribution over the C classes, and m = 1 2 (f θ (x) + f θ (x)), we can compute these measures as follows:

d MSE (f θ (x), f θ (x)) = 1 C C k=1 (f θ (x) k -f θ (x) k ) 2 (2.1) d KL (f θ (x), f θ (x)) = 1 C C k=1 f θ (x) k log f θ (x) k f θ (x) k (2.2) d JS (f θ (x), f θ (x)) = 1 2 d KL (f θ (x), m) + 1 2 d KL (f θ (x), m) (2.3)
Note that we can also enforce a consistency over two perturbed versions of x, x1 and x2 .

Denoising Decoder

Encoder Noisy Encoder

Gaussian Noise

Figure 2: Ladder Networks. An illustration of one forward pass of Ladder Networks. The objective is to reconstruct the clean activations of the encoder using a denoising decoder that takes as input the corrupted activations of the noisy encoder.

Ladder Networks

To take any well-performing feed-forward network on supervised data and augment it with additional branches to be able to utilize additional unlabeled data. Rasmus et al. [START_REF] Rasmus | Semi-supervised learning with ladder networks[END_REF] propose to use Ladder Networks [START_REF] Valpola | From neural pca to deep unsupervised learning[END_REF] with an additional encoder and decoder for SSL. As illustrated in fig. 2, the network consists of two encoders, a corrupted and clean one, and a decoder. At each training iteration, the input x is passed through both encoders. In the corrupted encoder, Gaussian noise is injected at each layer after batch normalization, producing two outputs, a clean prediction y and a prediction based on corrupted activations ỹ. The output ỹ is then fed into the decoder to reconstruct the uncorrupted input and the clean hidden activations. The unsupervised training loss L u is then computed as the MSE between the activations of the clean encoder z and the reconstructed activations ẑ (i.e., after batch normalization), computed over all layers, from the input to the last layer L, with a weighting λ l for each layer's contribution to the total loss:

L u = 1 |D| x∈D L l=0 λ l d MSE (z (l) , ẑ(l) ) (2.4)
If the input is a labeled data point, x ∈ D l , with a label y, a supervised cross-entropy loss H(ỹ, t) term can be added to L u to obtain the final loss.

L = L u + L s = L u + 1 |D l | x,t∈D l H(ỹ, t) (2.5)
The method can be easily adapted for convolutional neural networks (CNNs) by replacing the fullyconnected layers with convolutional layers for semi-supervised vision tasks. However, the ladder network is quite computationally heavy, approximately tripling the computation needed for one training iteration. To mitigate this, the authors propose a variant of ladder networks called Γ-Model where λ l = 0 when l < L. In this case, the decoder is omitted, and the unsupervised loss is computed as the MSE between the two outputs y and ỹ.

Network with dropout Augmentations

Cross Entropy Loss

Figure 3: Loss computation for Π-Model. The MSE between the two outputs is computed for the unsupervised loss, and if the input is a labeled example, we add the supervised loss to the weighted unsupervised loss.

Pi-Model

The Π-Model [START_REF] Laine | Temporal ensembling for semi-supervised learning[END_REF] is a simplification of the Γ-Model of Ladder Networks, where the corrupted encoder is removed, and the same network is used to get the prediction for both corrupted and uncorrupted inputs. Specifically, Π-Model takes advantage of the stochastic nature of the prediction function f θ in neural networks due to common regularization techniques, such as data augmentation and dropout, that typically don't alter the model predictions. For any given input x, the objective is to reduce the distances between two predictions of f θ with x as input in both forward passes. Concretely, as illustrated in fig. 3, we would like to minimize d(y, ỹ), where we consider one of the two outputs as a target. Given the stochastic nature of the predictions function (e.g., using dropout as a noise source), the two outputs f θ (x) = ỹ1 and f θ (x) = ỹ2 will be distinct, and the objective is to obtain consistent predictions for both of them. In case the input x is a labeled data point, we also compute the cross-entropy supervised loss using the provided labels y:

L = w 1 |D u | x∈Du d MSE (ỹ 1 , ỹ2 ) + 1 |D l | x,y∈D l H(y, f (x)) (2.6)
with w as a weighting function, starting from 0 up to a fixed weight λ (e.g., 30) after a given number of epochs (e.g., 20% of training time). This way, we avoid using the untrained and random prediction function, providing us with unstable predictions at the start of training.

Temporal Ensembling

Π-Model can be divided into two stages, we first classify all of the training data without updating the weights of the model, obtaining the predictions y, and in the second stage, we consider the predictions y as targets for the unsupervised loss and enforce consistency of predictions by minimizing the distance between the current outputs ỹ and the outputs of the first stage y under different dropouts and augmentations.

The problem with this approach is that the targets y are based on a single evaluation of the network and can rapidly change. This instability in the targets can lead to an instability during training and reduces the amount of training signal that can be extracted from the unlabeled examples. To solve this, Laine et al. [START_REF] Laine | Temporal ensembling for semi-supervised learning[END_REF] propose a second version of Π-Model called Temporal Ensembling, where the targets y ema are the aggregation of all the previous predictions. This way, during training, we only need a single forward pass to get the current predictions ỹ and the aggregated targets y ema , speeding up the training time by approximately 2×. The training process is illustrated in fig. 4.

For a target ỹ, at each training iteration, the current output ỹ is accumulated into the ensemble output y ema by an exponentially moving average update:

y ema = αy ema + (1 -α)ỹ (2.7)
where α is a momentum term that controls how far the ensemble reaches into training history. ỹ can also The MSE between the current prediction and the aggregated target is computed for the unsupervised loss, and if the input is a labeled example, we add the supervised loss to the weighted unsupervised loss.

be seen as the output of an ensemble network f from previous training epochs, with the recent ones having greater weight than the distant ones.

At the start of training, temporal ensembling reduces to Π-Model since the aggregated targets are very noisy, to overcome this, similar to the bias correction used in Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], the targets ỹ are corrected for the startup bias at a training step t as follows:

y ema = (αy ema + (1 -α)ỹ)/(1 -α t ) (2.8)
The loss computation in temporal ensembling remains the same as in Π-Model, but with two essential benefits. First, the training is faster since we only need a single forward pass through the network to obtain ỹ, while maintaining an exponential moving average (EMA) of label predictions on each training example and penalizing predictions that are inconsistent with these targets. Second, the targets are more stable during training, yielding better results. The downside of such a method is a large amount of memory needed to keep an aggregate of the predictions for all of the training examples, which can become quite memory intensive for large datasets and dense tasks (e.g., semantic segmentation).

Mean teachers

Π-Model and its improved version with Temporal Ensembling provides a better and more stable teacher model by maintaining an EMA of the predictions of each example, formed by an ensemble of the model's current version and those earlier versions evaluated at the same example. This ensembling improves the quality of the predictions and using them as the teacher predictions improve results. However, the newly learned information is incorporated into the training at a slow pace, since each target is updated only once per epoch, and the larger the dataset, the bigger the span between the updates gets.

Additionally, in the previous approaches, the same model plays a dual role, as a teacher and a student. Given a set of unlabeled data, as a teacher, the model generates the targets, which are then used by itself as a student for learning using a consistency loss. These targets may very well be misclassified. If the weight of the unsupervised loss outweighs that of the supervised loss, the model is prevented from learning new information, predicting the same targets, and resulting in a form of confirmation bias. To solve this, the quality of the targets must be improved. The quality of targets can be improved by either: (1) carefully choosing the perturbations instead of merely injecting additive or multiplicative noise, or, (2) carefully choosing the teacher model responsible for generating the targets, instead of using a replica of the student model.

To overcome these limitations, Mean Teacher [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] proposes using a teacher model for a faster incorporation of the learned signal, and to avoid the problem of confirmation bias. A training iteration of Mean Teacher (fig. 5) is very similar to previous methods; the main difference is that Π-Model uses the same model as a student and a teacher θ = θ, and Temporal Ensembling approximate a stable teacher f θ as an ensemble function with a weighted average of successive predictions. While Mean Teacher defines the weights θ t of the teacher model f θ at a training step t as an EMA of successive student's weights θ:

θ t = αθ t-1 + (1 -α)θ t (2.9)
The loss computation in this case is the sum of the supervised and unsupervised loss, where the teacher model is used to obtain the targets for the unsupervised loss for a given input x:

L = w 1 |D u | x∈Du d MSE (f θ (x), f θ (x)) + 1 |D l | x,y∈D l H(y, f θ (x))
(2.10)

Dual Students

One of the main drawbacks of using a Mean Teacher is that given a large number of training iterations, the teacher model weights will converge to that of the student model, and any biased and unstable predictions will be carried over to the student.

To solve this, Ke et al. [START_REF] Ke | Dual student: Breaking the limits of the teacher in semi-supervised learning[END_REF] propose a dual students step-up. Two student models with different initialization are simultaneously trained, and at a given iteration, one of them provides the targets for the other. To choose which one, we test for the most stable predictions that satisfy the following stability conditions:

• The predictions using two input versions, a clean x and a perturbed version x give the same results:

f (x) = f (x).
• Both predictions are confident, i.e., are far from the decision boundary. This can be tested by seeing if f (x) (resp. f (x)) is greater than a confidence threshold , e.g., = 0.1.

Given two student models, f θ1 and f θ2 , an unlabeled input x ∈ D u and its perturbed version x. We compute four predictions: f θ1 (x), f θ1 (x), f θ2 (x), and f θ2 (x) . In addition to training each model to minimize both the supervised and unsupervised losses:

L = L s + λ 1 L u = 1 |D l | x,y∈D l H(y, f θi (x)) + λ 1 1 |D u | x∈Du d MSE (f θi (x), f θi (x)) (2.11)
we also force one of the students to have similar predictions to its counterpart. To chose which one to update its weights, we check for both models' stability constraint; if the predictions one of the models is unstable, we update its weights. If both are stable, we update the model with the largest variation

E i = f i (x) -f i (x) 2 ,
i.e., the least stable. In this case, the least stable model is trained with an additional loss:

λ 2 x∈Du d MSE (f θi (x), f θj (x)) (2.12)
where λ 1 and λ 2 are hyperparameters specifying the contribution of each loss term.

Fast-SWA

Athiwaratkun et al. [START_REF] Athiwaratkun | There are many consistent explanations of unlabeled data: Why you should average[END_REF] observed that Π-Model and Mean Teacher continue taking significant steps in the weight space at the end of training, given that the models stochastic gradient descent (SGD) traverses a large flat region of the weight space late in training, continuing to actively explore the set of plausible solutions and producing diverse predictions on the test set even in the late stages of training. Based on this behavior, averaging the SGD iterates can lead to final weights closer to the center of the flat region, stabilizing the SGD trajectory, and leading to significant gains in performance and better generalization.

One way to produce an ensemble of the model late in training is Stochastic Weight Averaging (SWA) [START_REF] Izmailov | Averaging weights leads to wider optima and better generalization[END_REF], an approach based on averaging the weights traversed by SGD at the end of training with a cyclic learning rate (fig. 6). After a given number of epochs, the learning rate changes to a cyclic learning rate and the training repeats for several cycles, the weights at the end of each cycle corresponding to the minimum values of the learning rate are stored, and averaged together to obtain a model with the averaged weights f θ SWA , which is then used to make predictions. Motivated by the observation that the benefits of averaging are the most prominent when the distance between the averaged points is large, and given that SWA only collects the weights once per cycle, which means that many additional training epochs are needed in order to collect enough weights for averaging. The authors propose fast-SWA, a modification of SWA that averages the networks corresponding to many points during the same cycle, resulting in a better final model and a faster ensembling procedure.

Virtual Adversarial Training

The previous approaches focused on applying random perturbations to each input to generate artificial input points, encouraging the model to assign similar outputs to the unlabeled data points and their perturbed versions. This way, we push for a smoother output distribution. As a result, the generalization performance of the model can be improved. However, such random noise and random data augmentation often leaves the predictor particularly vulnerable to small perturbations in a specific direction, that is, the adversarial direction, which is the direction in the input space in which the label probability p(y|x) of the model is most sensitive.

To overcome this, and inspired by adversarial training [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] that trains the model to assign to each input data a label that is similar to the labels of its neighbors in the adversarial direction. Miyato et al. [START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semi-supervised learning[END_REF] propose Virtual Adversarial Training (VAT), a regularization technique that enhances the model's robustness around each input data point against random and local perturbations, the term virtual comes from the fact that the adversarial perturbation is approximated without any label information, and is hence applicable to SSL to smooth the output distribution.

Concretely, VAT trains the output distribution to be identically smooth around each data point, by selectively smoothing the model in its most adversarial direction. For a given data point x, we would like to compute the adversarial perturbation r adv that will alter the model's predictions the most. We start by sampling a Gaussian noise r of the same dimensions as the input x, we then compute its gradients grad r with respect the loss between the two predictions, with and without the injections of the noise r (i.e., KLdivergence is used as a distance measure d(., .)). r adv can then be obtained by normalizing and scaling grad r by a hyperparameter . The computation can be summarized in the following steps:

1. r ∼ N (0, ξ √ dim(x) I) 2. grad r = ∇ r d KL (f θ (x), f θ (x + r))
3. r adv = gradr gradr Note that the computation above is a single iteration of the approximation of r adv , for a more accurate estimate, we consider r adv = r and recompute r adv following the last two steps. But in general, given how computationally expensive this computation is, requiring an additional forward and backward passes, we only apply a single power iteration for computing the adversarial perturbation. With the optimal perturbation r adv , we can then compute the unsupervised loss as the MSE between the two predictions of the model, with and without the injection of r adv :

L u = w 1 |D u | x∈Du d MSE (f θ (x), f θ (x + r adv )) (2.13)
For a more stable training, a Mean Teacher can be used to generate stable targets by replacing f θ (x) with f θ (x), where f θ is an EMA of the student f θ .

Adversarial Dropout

Instead of using an additive adversarial noise as VAT, Park et al. [START_REF] Park | Adversarial dropout for supervised and semi-supervised learning[END_REF] propose adversarial dropout (AdD), a.k.a, element-wise adversarial dropout (EAdD), in which dropout masks are adversarially optimized to alter the model's predictions. With this type of perturbations, we induce a sparse structure of the neural network, while the other forms of additive noise does not make changes to the structure of the neural network directly.

Perturbed images

Clean images The first step is to find the dropout conditions that are most sensitive to the model's predictions. In a SSL setting, where we do not have access to the true labels, we use the model's predictions on the unlabeled data points to approximate the adversarial dropout mask adv , which is subject to the boundary condition:

adv -2 ≤ δH with H as the dropout layer dimension and a hyperparameter δ, which restricts adversarial dropout masks to be infinitesimally different from the random dropout mask . Without this constraint, the adversarial dropout might induce a layer without any connections. By restricting the adversarial dropout to be similar to the random dropout, we prevent finding such an irrational layer, which does not support backpropagation.

Similar to VAT, we start from a random dropout mask, we compute a KL-divergence loss between the outputs, with and without dropout, and given the gradients of the loss with respect to the activations before the dropout layer, we update the random dropout mask in an adversarial manner. The prediction function f θ is divided into two parts, f θ1 and f θ2 , where

f θ (x, ) = f θ2 (f θ1 (x)
), we start by computing an approximation of the Jacobian matrix as follows:

J(x, ) ≈ f θ1 (x) ∇ f θ 1 (x) d KL (f θ (x), f θ (x, )) (2.14)
Using J(x, ), we can then update the random dropout mask to obtain adv , so that if (i) = 0 and J(x, )(i) > 0 or (i) = 1 and J(x, )(i) < 0 at a given position i, we inverse the value of at that location. Resulting in adv , which can then be used to compute the unsupervised loss:

L u = w 1 |D u | x∈Du d MSE (f θ (x), f θ (x, adv )) (2.

15)

Channel-wise Adversarial Dropout The element-wise adversarial dropout (EAdD) introduced by Park et al. [START_REF] Park | Adversarial dropout for supervised and semi-supervised learning[END_REF] is limited to fully-connected networks, to use AdD in a wider range of tasks, Lee et al. [START_REF] Lee | Drop to adapt: Learning discriminative features for unsupervised domain adaptation[END_REF] proposed channel-wise AdD (CAdD), an extension the element-wise masking in AdD to convolutional layers (fig. 8). In these layers, standard dropout is relatively ineffective due to the strong spatial correlation between individual activations of a feature map [START_REF] Tompson | Efficient object localization using convolutional networks[END_REF]. EAdD dropout suffers from the same issues when naively applied to convolutional layers. To solve this, EAdD adversarially drops entire feature maps rather than individual activations. While the general procedure is similar to that of EAdD, an additional constraint is imposed on the mask to represent spatial dropout [START_REF] Tompson | Efficient object localization using convolutional networks[END_REF]. In this case, the mask ∈ R C×H×W is of the same shape as the activations; the adversarial dropout mask adv is approximated under the following new condition:

1 HW C i=1 adv (i) -(i) ≤ δC (2.16)
where δ is a hyperparameter to restrict the different between the two masks to be small, and (i) is the mask corresponding to the i-th activation map. The process of finding the channel-wise adversarial dropout mask is similar to those of element-wise adversarial dropout, but with a per activation map approximation.

Interpolation Consistency Training

As discussed earlier, the random perturbations are inefficient in high dimensions, given that only a limited subset of the input perturbations are capable of pushing the decision boundary into low-density regions. VAT and AdD find the adversarial perturbations that will maximize the change in the model's predictions, which involve multiple forward and backward passes to compute these perturbations. This additional computation can be restrictive in many cases and makes such methods less appealing. As an alternative, Verma et al. [START_REF] Verma | Interpolation consistency training for semi-supervised learning[END_REF] propose Interpolation Consistency Training (ICT) as an efficient consistency regularization technique for SSL.

Given a MixUp operation [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF]:

Mix λ (a, b) = λ • a + (1 -λ)
• b that outputs an interpolation between the two inputs with a weight λ ∼ Beta(α, α) for α ∈ [0, ∞]. As shown in fig. 9, ICT trains the prediction function f θ to provide consistent predictions at different interpolations of unlabeled data points x i and x j , where the targets are generated using a teacher model f θ which is an EMA of f θ :

f θ (Mix λ (x i , x j )) ≈ Mix λ (f θ (x i ), f θ (x j ))
(2.17)

The unsupervised objective is to have similar values between the student model's prediction given a mixed input of two unlabeled data points, and the mixed outputs of the teacher model.

L u = w 1 |D u | xi,xj ∈Du d MSE (f θ (Mix λ (x i , x j )), Mix λ (f θ (x i ), f θ (x j )) (2.18)
The benefit of ICT compared to random perturbations can be analyzed by considering the mixup operation as a perturbation applied to a given unlabeled example: x i + δ = Mix λ (x i , x j ), for a large number of classes and with a similar distribution of examples per class, it is likely that the pair of points (x i , x j ) lie in different clusters and belong to different classes. If one of these two data points lies in a low-density region, applying an interpolation toward x j points to a low-density region, which is a good direction to move the decision boundary toward.

Unsupervised Data Augmentation

Unsupervised Data Augmentation [START_REF] Xie | Unsupervised data augmentation for consistency training[END_REF] uses advanced data augmentation methods, such as AutoAugment [START_REF] Ekin D Cubuk | Autoaugment: Learning augmentation strategies from data[END_REF], RandAugment [START_REF] Ekin D Cubuk | Randaugment: Practical data augmentation with no separate search[END_REF] and Back Translation [START_REF] Edunov | Understanding back-translation at scale[END_REF][START_REF] Sennrich | Improving neural machine translation models with monolingual data[END_REF] Motivated by these points, Xie et al. [START_REF] Xie | Unsupervised data augmentation for consistency training[END_REF] propose to apply the following augmentations to generate transformed versions of the unlabeled inputs:

• RandAugment for Image Classification. Consists of uniformly sampling from the same set of possible transformations in Python Imaging Library (PIL), without requiring any labeled data to search for a good augmentation strategy.

• Back-translation for Text Classification. Consists of translating an existing example in language A into another language B, and then translating it back into A to obtain an augmented example.

After defining the augmentations to be applied during training, the training procedure (fig. 10) is straightforward. The objective is to have the correct predictions over the labeled set and consistent predictions on the original and augmented examples from the unlabeled set.

Entropy Minimization

In the previous section, in a setting where the cluster assumption is maintained, we enforce consistency of predictions to push the decision boundary into low-density regions to avoid classifying samples from the same cluster with distinct classes, which is a violation of the cluster assumption. Another way to enforce this is to encourage the network to make confident (i.e., low-entropy) predictions on unlabeled data regardless of the predicted class, discouraging the decision boundary from passing near data points where it would otherwise be forced to produce low-confidence predictions. This is done by adding a loss term which minimizes the entropy of the prediction function f θ (x). For a categorical output space with C possible classes, the entropy minimization term [START_REF] Grandvalet | Semi-supervised learning by entropy minimization[END_REF] is:

- C k=1 f θ (x) k log f θ (x) k (3.1)
However, with high capacity models such as neural networks, the model can quickly overfit to low confident data points by simply outputting large logits, resulting in a model with very confident predictions [START_REF] Oliver | Realistic evaluation of deep semi-supervised learning algorithms[END_REF]. On its own, entropy minimization doesn't produce competitive results compared to other SSL methods, but can produce state-of-the-art results when combined with different approaches.

Proxy-label Methods

Proxy label methods are the class of SSL algorithms that produce proxy labels on unlabeled data, using the prediction function itself or some variant of it without any supervision. These proxy labels are then used as targets together with the labeled data, providing some additional training information even if the produced labels are often noisy or weak and do not reflect the ground truth. These methods can be divided mainly into two groups: self-training, where the model itself produces the proxy labels, and multi-view learning, where the proxy labels are produced by models trained on different views of the data.

Self-training

In self-training [START_REF] Yarowsky | Unsupervised word sense disambiguation rivaling supervised methods[END_REF][START_REF] Scudder | Probability of error of some adaptive pattern-recognition machines[END_REF][START_REF] Riloff | Automatically generating extraction patterns from untagged text[END_REF][START_REF] Riloff | Learning extraction patterns for subjective expressions[END_REF], the small amount of labeled data D l is first used to train a prediction function f θ . The trained model is then used to assign pseudo-labels to the unlabeled data points x ∈ D u . Given an output f θ (x) for an unlabeled data point x in the form of a probability distribution over the classes, the pair (x, argmaxf θ (x)) is added to the labeled set if the probability assigned to its most likely class is higher than a predetermined threshold τ . The process of training the model using the augmented labeled set, and then set using it to label the remaining of D u is repeated until the model is incapable of producing confident predictions. Other heuristics can be used to decide which proxy labeled examples to retain, such as using the relative confidence instead of the absolute confidence, where the top n unlabeled samples predicted with the highest confidence after every epoch are added to the labeled training dataset D l . The impact of self-training is similar to that of entropy minimization; in both cases, the network is forced to output more confident predictions. The main downside of such methods is that the model is unable to correct its own mistakes, and any biased and wrong classifications can be quickly amplified resulting in confident but erroneous proxy labels on the unlabeled data points.

Yalnizet et al. [START_REF] Zeki Yalniz | Billion-scale semi-supervised learning for image classification[END_REF] propose to use self-training to improve ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] top-1 accuracy and enhance the robustness of the trained model to various perturbations (e.g., perturbations used in ImageNet-A, C and P [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF]). The model is first trained on unlabeled images and their proxy labels, and then fine-tuned on labeled images in the final stage. Instead of using the same model for both proxy labels generation and training, Xie et al. [START_REF] Xie | Self-training with noisy student improves imagenet classification[END_REF] propose to use the student-teacher setting. In an iterative manner, the teacher model is first trained on the labeled examples and used to generate soft proxy labels on the unlabeled data. The student can then be trained on both the labeled set and the proxy labels while aggressively injecting noise to obtain a more robust model. In the next iteration, the student is considered as a teacher, and a bigger version of EfficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] is used for the student, and the same procedure is repeated up to the largest model.

In addition to image classification, self-training was also successfully applied to a variety of tasks, such as semantic segmentation [START_REF] Babakhin | Semi-supervised segmentation of salt bodies in seismic images using an ensemble of convolutional neural networks[END_REF], text classification [START_REF] Li | Learning to self-train for semi-supervised few-shot classification[END_REF][START_REF] Karamanolakis | Leveraging just a few keywords for fine-grained aspect detection through weakly supervised co-training[END_REF], machine translation [START_REF] Sennrich | Improving neural machine translation models with monolingual data[END_REF][START_REF] He | Dual learning for machine translation[END_REF][START_REF] Cheng | Semi-supervised learning for neural machine translation[END_REF][START_REF] He | Revisiting self-training for neural sequence generation[END_REF] and when learning from noisy data [START_REF] Veit | Learning from noisy large-scale datasets with minimal supervision[END_REF].

Pseudo-labeling [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF][START_REF] Arazo | Pseudo-labeling and confirmation bias in deep semi-supervised learning[END_REF][START_REF] Iscen | Label propagation for deep semi-supervised learning[END_REF][START_REF] Shi | Transductive semi-supervised deep learning using min-max features[END_REF], similar to self-training, the objective of pseudo-labeling is to generate proxy labels to enhance the learning process. A first attempt at adapting pseudo-labeling [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF] for deep learning constrained the usage of the proxy labels to a fine-tuning stage after pretraining the network. Shi et al. [START_REF] Shi | Transductive semi-supervised deep learning using min-max features[END_REF] propose to adapt Transductive SSL [START_REF] Joachims | Transductive inference for text classification using support vector machines[END_REF][START_REF] Joachims | Transductive learning via spectral graph partitioning[END_REF][START_REF] Zhang | Fast and robust graph-based transductive learning via minimum tree cut[END_REF][START_REF] Wang | Progressive graph-based transductive learning for multi-modal classification of brain disorder disease[END_REF] by treating the labels of unlabeled examples as variables and trying to determine their optimal labels together with the optimal model parameters, by minimizing the proposed loss function through the iterative training process. The generated proxy labels are considered as hard labels for the unlabeled examples, an uncertainty weight is then introduced, with large weights for examples with distant k-nearest neighbors in the feature space, in additiont to two loss terms encouraging intra-class compactness and inter-class separation, and a consistency term between samples with different perturbations. Iscen et al. [START_REF] Iscen | Label propagation for deep semi-supervised learning[END_REF] integrated label-propagation [START_REF] Zhu | Learning from labeled and unlabeled data with label propagation[END_REF][START_REF] Whitney | Bootstrapping via graph propagation[END_REF][START_REF] Gong | Label propagation via teaching-to-learn and learning-to-teach[END_REF] within pseudo-labeling. The method alternates between training the network on the labeled examples and pseudo-labels and then leveraging the learned representations to build a nearest neighbor graph where label propagation is applied to refine the hard pseudo-labels. They also introduce two uncertainty scores, one for every sample based on the entropy of the output probabilities to overcome the unequal confidence in the predictions, and a per-class scored based class population to deal with class-imbalance. Arazo et al. [START_REF] Arazo | Pseudo-labeling and confirmation bias in deep semi-supervised learning[END_REF] showed that a naive pseudolabeling overfits to incorrect pseudo-labels due to the so-called confirmation bias, and demonstrate that MixUp [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing this bias.

Meta Pseudo Labels Given how important the heuristics used to select which the proxy labels to add to the training set, where a proper method could lead to a sizable gain. Pham et al. [START_REF] Pham | Meta pseudo labels[END_REF] propose to use the student-teacher setting, where the teacher model is responsible for producing the proxy labels based on an efficient meta-learning algorithm called Meta Pseudo Labels (MPL), which encourages the teacher to adjust the target distributions of training examples in a manner that improves the learning of the student model. The teacher is updated by policy gradients computed by evaluating the student model on a held-out validation set.

A given training step of MPL consists of two phases (fig. 11): • Phase 1: The Student learns from the Teacher. In this phase, given a single input example x ∈ D l , the teacher f θ produces a target class-distribution to train the student f θ , where the pair (x, f θ (x)) is shown to the student to update its parameters by back-propagating from the cross-entropy loss.

• Phase 2: The Teacher learns from the Student's Validation Loss. After the student updates its parameters in first step, its new parameter θ(t + 1) are evaluated on an example (x val , y val ) from the held-out validation dataset using the cross-entropy loss. Since the validation loss depends on θ via the first step, this validation cross-entropy loss is also a function of the teacher's weights θ . This dependency allows us to compute the gradients of the validation loss with respect to the teacher's weights, and then update θ to minimize the validation loss using policy gradients.

While the student's performance allows the teacher to adjust and adapt to the student's learning state, this signal alone is not sufficient to train the teacher since when the teacher has observed enough evidence to produce meaningful target distributions to teach the student, the student might have already entered a bad region of parameters. To overcome this, the teacher is also trained using the pair of labeled data points from the held-out validation set.

Multi-view training

Multi-view training (MVL) [START_REF] Kumar | A co-training approach for multi-view spectral clustering[END_REF][START_REF] Zhao | Multi-view learning overview: Recent progress and new challenges[END_REF] utilizes multi-view data that are very common in real-world applications, where different views can be collected by different measuring methods (e.g., color information and texture information for images) or by creating limited views of the original data. In such a setting, MVL aims to learn a distinct prediction function f θi to model a given view v i (x) of a data point x, and jointly optimize all the functions to improve the generalization performance. Ideally, the possible views complement each other so that the produced models can collaborate in improving each other's performance.

Co-training

Co-training [START_REF] Blum | Combining labeled and unlabeled data with co-training[END_REF] requires that each data point x can be represented using two conditionally independent views v 1 (x) and v 2 (x), and that each view is sufficient to train a good model. After training two prediction functions f θ1 and f θ2 on a specific view on the labeled set D l . We start the proxy labeling procedure. At each iteration, an unlabeled data point is added to the training set of the model f θi if the other model f θj outputs a confident prediction with a probability higher than a threshold τ . This way, one of the models provides newly labeled examples where the other model is uncertain. Co-training has been combined with deep learning for some applications, such as object recognition [START_REF] Cheng | Semi-supervised multimodal deep learning for rgb-d object recognition[END_REF] by utilizing RGB-D data, with RGB and depth as the two views used to train the two models, or for combining multi-modal data [START_REF] Mohammady | Co-training for demographic classification using deep learning from label proportions[END_REF] (i.e., image and text) by training each model on a given modality and use it to provide pseudo-labels for other models. However, in many cases, the data have only one view rather than two, in this instance, different learning algorithms or different parameter configurations to learn two different classifiers can be employed. The two views v 1 (x) and v 2 (x) can also be generated by injecting noise or by applying different augmentations, for example, Qiao et al. [START_REF] Qiao | Deep co-training for semi-supervised image recognition[END_REF] used adversarial perturbations to produce new views for deep co-training for image classification, where the models are encouraged to have the same predictions on D l but make different errors when they are exposed to adversarial attacks.

Democratic Co-training [START_REF] Zhou | Democratic co-learning[END_REF]. An extension of Co-training, consists of replacing the different views of the input data with a number of models with different architectures and learning algorithms, which are first trained on the labeled examples. The trained models are then used to label a given example x if a majority of models confidently agree on its label.

Tri-Training

Tri-training [START_REF] Zhou | Tri-training: Exploiting unlabeled data using three classifiers[END_REF] tries to overcome the lack of data with multiple views and reduce the bias of the predictions on unlabeled data produced with self-training by utilizing the agreement of three independently trained models instead of a single model. First, the labeled data D l is used to train three prediction functions: f θ1 , f θ2 and f θ3 . An unlabeled data point x ∈ D u is then added to the supervised training set of the function f θi if the other two models agree on its predicted label. The training stops if no data points are being added to any of the models' training sets. Tri-training requires neither the existence of multiple views nor unique learning algorithms, making it more generally applicable. Using Tri-training with neural networks can be very expensive, requiring predictions for each one of the three models on all the unlabeled data. Ruder et al. [START_REF] Ruder | Strong baselines for neural semi-supervised learning under domain shift[END_REF] propose to sample a limited number of unlabeled data points at each training epoch, the candidate pool size is increased as the training progresses and the models become more accurate. [START_REF] Ruder | Strong baselines for neural semi-supervised learning under domain shift[END_REF] can also be used to reduce the time and sample complexity, where all three models share the same feature-extractor with model-specific classification layers. This way, the models are trained jointly with an additional orthogonality constraint on two of the three classification layers to be added to loss term, to avoid learning similar models and falling back to the standard case of self-training. Tri-Net [START_REF] Dong-Dongchen | Tri-net for semi-supervised deep learning[END_REF] also falls in this category, with a shared module for joint learning and three output modules for tri-training, in addition to utilizing output smearing [START_REF] Breiman | Randomizing outputs to increase prediction accuracy[END_REF] to initialize these modules. After the proxy labeling iteration, a fine-tuning stage is conducted on the labeled data to augment diversity and eliminate unstable and suspicious pseudo-labeled data.

Multi-task tri-training

Cross-View Training

In self-training, the model plays a dual role of a teacher and a student, producing the predictions it is being trained on, resulting in very moderate performance gains. As a solution, and taking inspiration from multi-view learning and consistency training, Clark et al. [START_REF] Clark | Semi-supervised sequence modeling with cross-view training[END_REF] propose Cross-View Training, where the model is trained to produce consistent predictions across different views of the inputs. Instead of using a single model as a teacher and a student, they propose to use a shared encoder, and then add auxiliary prediction modules that transform the encoder representations into predictions, these modules are then divided into auxiliary student modules and a primary teacher module. The input to each student prediction module is a subset of the model's intermediate representations corresponding to a restricted view of the input, such as feeding one of the student only the forward LSTM from a given Bi-LSTM layer, so it makes predictions without seeing any tokens to the right of the current one (fig. 12). The primary teacher module in trained only on labeled examples, and is responsible of generating the pseudo-labels taking as input the full view of the unlabeled inputs, the students are trained to have consistent predictions with the teacher module. Given an encoder e, a teacher module t and K student modules s i with i ∈ [0, K], where each student receives a limited view of the input, the training objective is written as follows:

L = L u + L s = 1 |D u | x∈Du K i=1 d MSE (t(e(x)), s i (e(x))) + 1 |D l | x,y∈D l H(t(e(x)), y) (4.1)
x 1 The student prediction modules can learn from the teacher module predictions because this primary module has a better, unrestricted view of the inputs. As the student modules learn to make accurate predictions despite their restricted views of the input, they improve the quality of the representations produced by the encoder. Which, in turn, improves the full model, which uses the same shared representations.

Holistic Methods

An emerging line of work in SSL is a set of holistic approaches that try to unify the current dominant methods in SSL in a single framework, achieving better performances.

MixMatch

Berthelot et al. [START_REF] Berthelot | Mixmatch: A holistic approach to semi-supervised learning[END_REF] propose a holistic approach which gracefully unifies ideas and components from the dominant paradigms for SSL, resulting in an algorithm that is greater than the sum of its parts and surpasses the performance of the traditional approaches.

MixMatch takes as input a batch from the labeled set D l containing pairs of inputs and their corresponding one-hot targets, a batch from the unlabeled set D u containing only unlabeled data, and a set of hyperparameters: the sharpening softmax temperature T , the number of augmentations K, and the Beta distribution parameter α for MixUp. Producing a batch of augmented labeled examples and a batch of augmented unlabeled examples with their proxy labels. These augmented examples can then be used to compute the losses and train the model. Precisely, MixMatch consists of the following steps:

• Step 1: Data Augmentation. Using a given transformation, a labeled example x ∈ D l from the labeled batch is transformed, producing its augmented versions x. For an unlabeled example x ∈ D u , the augmentation function is applied K times, resulting in K augmented versions of the unlabeled examples x1 , ..., xK .

• Step 2: Label Guessing. The second step consists of producing proxy labels for the unlabeled examples. First, we generate the predictions for the K augmented versions of each unlabeled example using the predictions function f θ . The K predictions are then averaged together, obtaining a proxy or a pseudo label ŷ = 1/K K k=1 (ŷ k ) for each one of the augmentations of the unlabeled example x: (x 1 , ŷ), ..., (x K , ŷ).

• Step 3: Sharpening. To push the model to produce confident predictions and minimize the entropy of the output distribution, the generated proxy labels ŷ in step 2 in the form of a probability distribution over C classes are sharpened by adjusting the temperature of the categorical distribution, computed as follows where (ŷ u ) k refers to the probability of class k out of C classes:

(ŷ) k = (ŷ) 1 T k / C k=1 (ŷ) 1 T k (5.1)
• Step 4 MixUp. The previous steps resulted in two new augmented batches, a batch L of augmented labeled examples and their target, and a batch U of augmented unlabeled examples and their sharpened proxy labels. Note that the size of U is K times larger than the original batch given that each example x ∈ D u is replaced by its K augmented versions. In the last step, we mix these two batches. First, a new batch merging both batches is created W = Shuffle(Concat(L, U)). W is then divided into two batches: W 1 of the same size as L and W 2 of the same size as L. Using the Mixup operation that is slightly adjusted so that the mixed example is closer the labeled examples, the final step is to create new labeled and unlabeled batches by mixing the produced batches together using Mixup as follows:

L = MixUp(L, W 1 ) (5.2)
U = MixUp(U, W 2 ) (5.3)
After creating two augmented batches L and U using MixMatch, we can then train the model using the standard SSL losses by computing the CE loss for the supervised loss, and the consistency loss for the unsupervised loss using the augmented batches as follows:

L = L s + wL u = 1 |L | x,y∈L H(y, f θ (x))) + w 1 |U | x,ŷ∈U d MSE (ŷ, f θ (x))
(5.4)

ReMixMatch

Berthelot et al. [START_REF] Berthelot | Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring[END_REF] propose to improve MixMatch by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly augmented versions of the input into the model, encouraging each output to be close to the prediction for a weakly-augmented version of the same input. Augmentation Anchoring. MixMatch uses a simple flip-and-crop augmentation strategy, ReMixMatch replaces the weak augmentations with strong augmentations learned using a control theory based augmentation strategy following AutoAugment. With such augmentations, the model's prediction for a weakly augmented unlabeled image is used as a proxy label for many strongly augmented versions of the same image in a standard cross-entropy loss.

For training, MixMatch is applied to the unlabeled and labeled batches, with the application of distribution alignment and replacing the K weakly augmented example with a strongly augmented example, in addition to using the weakly augmented examples for predicting proxy labels for the unlabeled strongly augmented examples. With two augmented batches L and U , the supervised and unsupervised losses are computed both using the cross-entropy loss as follows:

L = L s + wL u = 1 |L | x,y∈L H(y, f θ (x))) + w 1 |U | x,ŷ∈U H(ŷ, f θ (x)))
(5.5)

In addition to these losses, the authors add a self-supervised loss.

L SL = w 1 | Û | x,ŷ∈ Û H(ŷ, f θ (x))) + λ 1 | Û | x∈ Û H(r, f θ (x))) (5.6)

FixMatch

FixMatch [START_REF] Sohn | Fixmatch: Simplifying semi-supervised learning with consistency and confidence[END_REF] proposes a simple SSL algorithm that combines consistency regularization and pseudolabeling. In FixMatch (fig. 15), both the supervised and unsupervised losses are computed using a crossentropy loss. For labeled examples, the provided targets are used. For unlabeled examples x ∈ D u , a weakly augmented version is first computed using weak augmentation function A w . As in self-training, the predicted Figure 15: FixMatch. The model prediction on a weakly augmented input is considered as target if the maximum output class probability is above threshold, this target can then be used to train the model on a strongly augmented version of the same input using standard cross-entropy loss. Image Source: [START_REF] Berthelot | Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring[END_REF].

label is then considered as a proxy label if the highest class probability is greater than a threshold τ . With a proxy label, K strongly augmented examples are generated using a strong augmentation function A s . We then assign to these augmented versions the proxy label obtained with the weakly labeled version. The unsupervised loss can be written as follows:

L u = w 1 K|D u | x∈Du K i=1 1(max(f θ (A w (x))) ≥ τ )H(f θ (A w (x)), f θ (A s (x)))
(5.7)

Augmentations. Weak augmentations consist of a standard flip-and-shift augmentation strategy. Specifically, the images are flipped horizontally with a probability of 50% on all datasets except SVHN, in addition to randomly translating images by up to 12.5% vertically and horizontally. For the strong augmentations, RandAugment and CTAugment [START_REF] Berthelot | Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring[END_REF] are used where a given transformation (e.g., color inversion, translation, contrast adjustment, etc.) is randomly selected for each sample in a batch of training examples, and the amplitude of the transformation is a hyperparameter that is optimized during training.

Other important factors in the FixMatch are the usage of Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF], weight decay regularization and the learning rate schedule, where the authors propose to use a cosine learning rate decay with a decay of η cos( 7πt 16T ), where η is the initial learning rate, t is the current training step, and T is the total number of training iterations.

Generative Models

In unsupervised learning, we are provided with samples x drawn i.i.d. from an unknown data distribution with density p(x), and the objective is to estimate this density. Supervised learning, on the other hand, consists of estimating a functional relationship between the inputs x and the labels y with the goal of minimizing the functional of the joint distribution p(x, y) [START_REF] Chapelle | Semi-supervised learning[END_REF]. Classification can be treated as a special case of estimating p(x, y), where we are only interested in the conditional distributions p(y|x), without the need to estimate the input distribution p(x) since x will always be given at prediction time. Semi-supervised learning with generative models can be viewed as either an extension of supervised learning, classification in addition to information about p(x) provided by D u , or as an extension of unsupervised learning, clustering in addition to the provided labels from D l . In this section, we explore some generative approaches for deep SSL.

Variational Autoencoders for SSL

Variational Autoencoders (VAEs) [START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Doersch | Tutorial on variational autoencoders[END_REF] have emerged as one of the most popular approaches to unsupervised learning of complicated distributions. A standard VAE is an autoencoder trained with a reconstruction objective between the inputs and their reconstructed versions, in addition to a variational objective term that attempts to learn a latent space that roughly follows a unit Gaussian distribution, this objective is implemented as the KL-divergence between the latent space and the standard Gaussian. With an input x, the conditional distribution q φ (z|x) modeled by an encoder, the standard Gaussian distribution p(z) and the reconstructed input x generated using a decoder p θ (x|z). The parameters φ and θ are trained to minimize the following objective: L = d MSE (x, x) + d KL (q φ (z|x), p(z)) (6.1)

Variational Autoencoder

Kingma et al. [START_REF] Durk P Kingma | Semi-supervised learning with deep generative models[END_REF] expanded the work on variational generative techniques [START_REF] Diederik | Auto-encoding variational bayes[END_REF][START_REF] Jimenez Rezende | Stochastic backpropagation and approximate inference in deep generative models[END_REF] for SSL, that exploit generative descriptions of the data to improve upon the classification performance that would be obtained using the labeled data alone.

Standard VAEs for SSL (M1 Model)

The first model consists of an unsupervised pretraining stage, in which the VAE is trained using the labeled and unlabeled examples. Using a fully trained VAE, the observed labeled data x ∈ D l are transformed into the latent space defined by z, the standard supervised task can then be solved using (z, y) where y are the labels of x. With this approach, the classification can be performed in a lower dimensional space since the dimensionality of the latent variables z is much less than that of the observations. These low dimensional embeddings are more easily separable since the latent space is formed by independent Gaussian posteriors parameterized by an encoder, built by a sequence of non-linear transformations of the inputs.

Extending VAEs for SSL (M2 Model) In the M1 model, the labels of D l were ignored when training the VAE. With the second model, the labels are also used during training. If the class labels are not available, y is treated as a latent variable in addition to the latent variable z. The network in this case contains three components, q φ (y|x) modeled by a classification network, q φ (z|y, x) modeled by an encoder, and p θ (x|y, z) modeled by a decoder, with parameters φ and θ. The training is similar to a standard VAE with the addition of the posterior on y and loss terms to train q φ (y|x) if the labels are available. The distribution q φ (y|x) can then be used at test time to get the predictions on unseen data.

Stacked VAEs (M1+M2 Model)

The two previous models can be concatenated to form a joint model. In this case, the model M1 is first trained to obtain the latent variables z 1 , the model M2 then uses the latent variables z 1 from model M1 as new representations of the data as opposed to raw values x. The final model can be described as follows:

p θ (x, y, z 1 , z 2 ) = p(y)p(z 2 )p θ (z 1 |y, z 2 )p θ (x|z 1 ) (6.2)

Variational Auxiliary Autoencoder

Variational Auxiliary Autoencoder [START_REF] Maaløe | Auxiliary deep generative models[END_REF][START_REF] Ranganath | Hierarchical variational models[END_REF] extends the variational distribution with auxiliary variables a: q(a, z|x) = q(z|a, x)q(a|x), such that the marginal distribution q(z|x) can fit more complicated posteriors p(z|x) while improving the flexibility of inference. In order to have an unchanged generative model p(x|z), it is required that the joint mode p(x, z, a) gives back the original p(x, z) under marginalization over a, thus p(x, z, a) = p(a|x, z)p(x, z), with p(a|x, z) = p(a) to avoid falling back to the original VAE model.

In SSL, to incorporate the class information, an additional latent variable y is introduced, the generative model become p(y)p(z)p(a|z, y, x)p(x|y, z), with a, y, z as the auxiliary variable, class label, and latent features respectively. In this case, the auxiliary unit a introduces a latent feature extractor to the inference model giving a richer mapping between x and y. The resulting model is parametrized by 5 neural networks: 1) an auxiliary inference model q(a|x), 2) a latent inference model q(z|a, y, x), 3) a classification model q(y|a, x), 4) a generative model p(a|.), and 5) a generative model p(x|.), which are trained on both a generative and discriminative tasks simultaneously.

Infinite Variational Autoencoder

Another variation of VAEs for SSL is Infinite Variational Autoencoder [START_REF] M Ehsan Abbasnejad | Infinite variational autoencoder for semi-supervised learning[END_REF], to overcome the limitation of VAEs of having a fixed dimension of the latent space and a fixed number of parameters in the generative model in advance, in which the capacity of the model must be chosen a priori with some foreknowledge of the training data characteristics. Infinite VAE solves this by producing an infinite mixture of autoencoders capable of growing with the complexity of the data to best capture its intrinsic structure. After training the generative model using unlabeled data, this model can then be combined with the available labeled data to train a discriminative model, which is also a mixture of experts, for classification. For a given test example x, each discriminative expert produces a tentative output that is then weighted by the generative model. As such, each discriminative expert learns to perform better with instances that are more structurally similar from the generative model's perspective. With a higher modeling capability, the infinite VAE is able to capture the distribution of the unlabeled data more accurately. Therefore, it provides a generative model that allows the discriminative model, which is trained based on its output, to be more effectively learned using a small number of samples.

Generative Adversarial Networks for SSL

A Generative Adversarial Network (GAN) [START_REF] Goodfellow | Generative adversarial nets[END_REF] (fig. 16) consists of a generator network G and a discriminator network D. The generator receives a latent variable z ∼ p(z) sampled from the prior distribution p(z) and maps to the input space. The discriminator takes an input, either coming from the real data p(x) or generated by G and outputs the probability of the input being from either G or the real data distribution p(x), represented with an empirical distribution D. The standard training procedure of GANs minimizes two objectives by alternating between training the discriminator D and the generator G:

L D = max D E x∼p(x) [log D(x)] + E z∼p(z) [1 -log D(G(z))] L G = min G -E z∼p(z) [log D(G(z))] (6.3)
where p(z) is usually chosen as a standard normal distribution. Other formulations have been proposed to improve and stabilize the training procedure, such as the hinge-loss version of the adversarial loss [START_REF] Lim | Geometric gan[END_REF][START_REF] Tran | Deep and hierarchical implicit models[END_REF] and Wassertein GAN (WGAN) [START_REF] Arjovsky | Wasserstein gan[END_REF]. Which are subsequently improved in several ways [START_REF] Zhang | Self-attention generative adversarial networks[END_REF][START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF][START_REF] Zhang | Consistency regularization for generative adversarial networks[END_REF], such as using spectral normalization [START_REF] Miyato | Spectral normalization for generative adversarial networks[END_REF] on both the generator and the discriminator, or consistency regularization on the discriminator [START_REF] Zhang | Consistency regularization for generative adversarial networks[END_REF].

CatGAN

Categorical generative adversarial networks (CatGAN) [START_REF] Tobias | Unsupervised and semi-supervised learning with categorical generative adversarial networks[END_REF] consist of combining both the generative and the discriminative perspectives within the training procedure. The discriminator D in this case plays the role of C classifiers and is trained to maximize the mutual information between the inputs x and the predicted labels for a number of C unknown classes. To aid these classifiers in their task of discovering categories that generalize well to unseen data, and avoid overfitting to spurious correlations in the data, the adversarial generative network comes into play and provides the examples the discriminator must become robust to.

The traditional two-player game in the GAN framework can be extended to CatGAN by having a discriminator that assign all examples to one the C classes instead of a probability of x belonging to p(x), while staying uncertain of the class assignments for the generated samples by G. After training such a classifiergenerator pair where the discovered C classes coincide with the classification problem we are interested in, the classifier can then be used during inference being trained only on unlabeled data.

CatGAN objective dictates three requirements for the discriminator and two requirements that the generator should fulfilled:

• Discriminator requirements: should (1) be certain of class assignment for samples from p(x), (2) be uncertain of assignment for generated samples, and (3) by assuming a uniform prior p(y) over classes, all classes must be distributed equally.

• Generator requirements: should (1) generate samples with highly certain class assignments, and (2) similar to the discriminator, equally distribute samples across all classes.

In order to have the output class distribution D(x) = p(y|x, D) to be highly peaked where D is certain about the class assignment, the entropy H(D(x)) of the class distribution must be low. For the generated samples D(G(z)) = p(y|G(z), D), the predictions should be highly uncertain with a uniform class distribution, in this case, the entropy H(D(G(z))) must be high. The first two requirements can then be enforced by simply minimizing H(D(x)) and maximizing the H(D(G(z))). To meet the third requirement that all classes should be used equally, the entropy of the marginal class distribution as measured empirically for both D and G needs to be maximized:

H D = H 1 N N i=1 D(x i ) H G ≈ H 1 M M i=1 D(G(z i )) (6.4)
Combining these requirements, CatGAN objective for the discriminator and the generator is:

L D = max D -E x∼p(x) [H(D(x))] + E z∼p(z) [H(D(G(z)))] + H D L G = min G E z∼p(z) [H(D(G(z)))] -H G (6.5)
In SSL, if the input x comes from the labeled set D l with a label y in the form of a one-hot vector, the discriminator D is trained with a cross-entropy loss in addition to L D :

L D + λE (x,y)∼p(x) l [-y log G(x)] (6.6)
where λ is a cost weighting term.

DCGAN

Another way of using GANs for SSL is to leverage the unlabeled examples to learn good and transferable intermediate representations, which can then be used on a variety of supervised learning tasks such as image classification based on a small labeled set D l . Radford et al. [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF] propose to build good image representations by training GANs, and later reusing parts of the generator and discriminator networks as feature extractors for supervised tasks. The authors propose Deep Convolutional GANs (DCGAN), a class of architectures with a set of constraints on the architectural topology of convolutional GANs to be able to scale them while maintaining a stable training in most settings, such as replacing polling layers with strided convolutions for the discriminator, fractional-strided convolutions for generator, using batchnorm [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] in both the generator and the discriminator, and removing fully connected layers for deeper architectures.

After training DCGANs for image generation, the representations learned by DCGANs can be utilized for downstream tasks, by either fine tuning the discriminator features with an additional classification layer added on top and trained on D l , or by flattening and concatenating the learned features and training a linear classifier on top of them.

SGAN

DCGAN demonstrated the utility of the learned representations for SSL, but it has several undesirable properties. Using the learned representations of the discriminator after the fact doesn't allow for training the classifier and the generator simultaneously, doing this is more efficient, but more importantly, improving the discriminator improves the classifier, and improving the classifier improves the discriminator, which improves the generator. Semi-Supervised GAN (SGAN) [START_REF] Odena | Semi-supervised learning with generative adversarial networks[END_REF] takes advantage of this feedback loop by allowing to learn a generative model and a classifier simultaneously, significantly improving the classification performance, the quality of the generated samples, and reducing training time.

Instead of a discriminator network outputting an estimated probability that the input image is drawn from the data distribution. For C classes, SGAN consists of a discriminator with C + 1 output, with per class output in addition to a fake class output. Training an SGAN is similar to training a GAN; the only difference is using the labels to train the discriminator if the input x is drawn for the labeled set D l . The discriminator is trained to minimize the negative log-likelihood with respect to the given labels, and the generator is trained to maximize it.

Feature Matching GAN

Training GANs consists if finding a Nash equilibrium to a two-player non-cooperative game, with each player trying to minimize its cost function. To solve this, GAN training consists of applying gradient descent on each player's cost simultaneously, but with such a training procedure, there is no guarantee of convergence. Feature matching [START_REF] Salimans | Improved techniques for training gans[END_REF] was proposed to encourage convergence. Feature matching addresses the instability of GANs by specifying a new objective for the generator that prevents it from over training on the current discriminator. Instead of directly maximizing the output of the discriminator, the new objective requires the generator to generate data that matches the first-order feature statistics between of the data distribution, i.e., the hidden representations of the discriminator. For some activations h(x) of a given intermediate layer, the new objective is defined as:

E x∼p(x) [h(x)] -E z∼p(z) [h(G(z))] 2 (6.7)
The problem of the generator mode collapse, where it always emits the same point, is still present even with feature matching because the discriminator processes each example independently, so there is no coordination between its gradients, and thus no mechanism to tell the outputs of the generator to become more dissimilar to each other. To avoid this, in addition to feature matching, a new technique called minibatch discrimination is also integrated into the training procedure to allow the discriminator to look at multiple data examples in combination, where the discriminator still classifies single examples as real or generated data, but it is now able to use the other examples in the minibatch as side information.

For SSL, similar to SGAN, the discriminator in feature matching GAN employs a (C + 1)-class objective instead of binary classification, where true samples are classified into the first C classes and generated samples are classified into the (C +1)-th fake class, the probability of x being fake in this case is p(y = C +1|G(z), D), corresponding to 1 -D(x) in the original GAN framework. The loss function for training the classifier then becomes L = L s + L u where:

L s = -E x,y∼p(x) l [log p(y|x, y < K + 1, D)] L u = -E x∼p(x)u log[1 -p(y = K + 1|x, D)] -E z∼p(z) log[p(y = K + 1|G(z), D))] (6.8)
The above objective is similar to the original GAN formulation by considering p(y = K + 1|G(z), D) to be the probability of fake samples, while the only difference is that the probability of true samples if split into C sub-classes. This (C + 1)-class discriminator objective lead to strong empirical results, and was later widely used to evaluate the effectiveness of generative models [START_REF] Vincent Dumoulin | Adversarially learned inference[END_REF][START_REF] Ulyanov | It takes (only) two: Adversarial generator-encoder networks[END_REF]. The main drawback is that feature matching works well in classification but fails to generate indistinguishable samples, while the other objective of minibatch discrimination is good at realistic image generation but cannot predict labels accurately.

Bad GAN

Feature matching GAN formulation raises two questions. First, it is not clear why the formulation of the discriminator can improve the performance when combined with a generator. Second, it seems that good semi-supervised learning and a good generator cannot be obtained at the same time. Dai et al. [START_REF] Dai | Good semi-supervised learning that requires a bad gan[END_REF] addressed these questions by showing that for a (C + 1)-class discriminator formulation of GAN-based SSL, good semisupervised learning requires a bad generator that does not match the true data distribution, but simply plays the role of a complement generator to help the discriminator obtain correct decision boundaries in high-density areas in the feature space.

To overcome the drawbacks of feature matching GANs, the new objective function of the generator is:

min G -H(p G ) + E x∼p G log p(x)I[p(x) > ] + E x∼p G h(x) -E x∼p(x) h(x) 2 (6.9)
where p G is the distribution induced by the generator G, I[•] is an indicator function and is a threshold. The first term maximizes the entropy of generator to avoid the collapsing issues that are a clear sign of low entropy, but given that for implicit generative models, GANs only provide samples rather than an analytic density form, the entropy can either optimized in the input space i.e., H(p G (x)) using variational inference or the feature space i.e., H(p G (h(x))) using a pull-away term (PT) [START_REF] Zhao | Energy-based generative adversarial network[END_REF] as an auxiliary cost for the entropy. The second term enforces the generation of samples with low density in the input space by pushing the generated samples to move towards low-density regions defined by p(x), this probability distribution over images is estimated using PixelCNN++ [START_REF] Salimans | Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications[END_REF] model, which pretrained on the training set, and fixed during semi-supervised training. The last term is the feature matching objective. This method substantially improves the performance of image classification over vanilla feature matching GANs on several benchmark datasets.

Triple-GAN

As discussed in Bad GAN, the generator and the discriminator (i.e., the classifier) may not be optimal at the same time, since that for an optimal generator, i.e., p(x) = p g (x), an optimal discriminator should identify x as fake. Still, as a classifier, the discriminator should predict the correct class of x confidently since x ∼ p(x), indicating that the discriminator and generator may not be optimal at the same time. Instead of learning a complement generator for classification, Triple-GAN [START_REF] Li Chongxuan | Triple generative adversarial nets[END_REF] is designed to achieve simultaneously a good generation of realistically-looking samples conditioned on class labels, and produce a good classifier with the smallest possible prediction error.

Triple-GAN consists of three components: (1) a classifier C that characterizes the conditional distribution p c (y|x) ≈ p(y|x); (2) a class-conditional generator G that characterizes the conditional distribution in the other direction p g (x|y) ≈ p(x|y); and (3) a discriminator D that distinguishes whether a pair of data (x, y) comes from the true distribution p(x, y). All the components are parameterized as neural networks. The desired equilibrium is that the joint distributions defined by the classifier and the generator both converge to the true data distribution.

For p(x) as the empirical distribution of inputs x ∈ D and p(y) as a uniform distribution which is assumed to be the same as the distribution of labels on labeled data, the classifier produces pseudo-labels p c (y|x) given x, in this case, the examples x and the pseudo-labels y are drown from the joint distribution p c (x, y) = p(x)p c (y|x). Similarly, the generator produces examples x = G(y, z), with y ∼ p(y) and the latent variables z ∼ p(z), the generated examples x and labels y are drown from the joint distribution p g (x, y) = p(y)p g (x|y). These pseudo input-label pairs (x, y) generated by both C and G are sent to the single discriminator D. The objective function is formulated as: where α ∈ [0, 1] is a constant that controls the relative importance of generation and classification. To properly leverage unlabeled data, an additional regularization is enforced on classifier C, consisting of minimizing the conditional entropy of p c (y|x), the cross-entropy between p(y) and p c (y), and a consistency regularization with a dropout as the source of noise. In such a setting, the classifier achieves high accuracy with only very few labeled examples, while the generator produces state-of-the-art images, even when conditioned on y labels.

L = min
Enhanced TripleGAN (EnhancedTGAN) [START_REF] Wu | Enhancing triplegan for semi-supervised conditional instance synthesis and classification[END_REF] improves Triple-GAN by adopting a class-wise mean feature matching to regularize the generator and a semantic matching term to ensure the semantics consistency of the synthesized data between the generator and the classifier, further improving the state-of-the-art results in both SSL and instance synthesis.

BiGAN

One of the limitations of the traditional GAN framework is not being able to infer latent representations z that can be used as rich representations of the data x for a more efficient training. Unlike VAEs with an inference network (i.e., decoder) p(.) that can learn a variational posterior over latent variables, the generator is typically a directed, latent variable model with latent variables z and observed variables x, making it unable to infer the latent feature representations for a given data point. BiGAN [START_REF] Donahue | Adversarial feature learning[END_REF] solves this by introducing an encoder E as an additional component in the GAN framework, which maps data x to latent representations z. The BiGAN discriminator D discriminates not only in data space between x and G(z), but jointly in data and latent space, between pairs (x, E(x)) and (G(z), z), where the latent component is either an encoder output E(x) or a generator input z. A trained BiGAN encoder can then serve as feature extractor for downstream tasks. The BiGAN training objective is defined as a minimax objective:

L = min G,E max D E x∼p(x) (log D(x, E(x))) + E z∼p(z) (1 -log D(G(z), z)) (6.11)
Kumar et al. [START_REF] Kumar | Semi-supervised learning with gans: Manifold invariance with improved inference[END_REF] proposed Augmented-BiGAN, an improved version of BiGAN for SSL. The Augmented-BiGAN is similar to other GAN frameworks used for SSL, treating the generated samples as an additional class to the regular classes that the classifier aims to label, with an additional Jacobian-based regularization that is introduced to encourage the classifier to be robust to local variations in the tangent space of the input manifold. The BiGAN trained encoder is used in calculating these Jacobians, resulting in an efficient estimation of the tangents space at each training sample, and avoiding the expensive SVD-based method used in contractive autoencoders [START_REF] Rifai | The manifold tangent classifier[END_REF].

7 Graph-Based SSL Graphs are a powerful tool to model interactions and relations between different entities, in order to understand the represented system in both a global and local manner. In Graph-based SSL [START_REF] Xiaojin | Semi-supervised learning literature survey[END_REF] methods, each data point x i , be it labeled or unlabeled, is represented as a node in the graph, and the edge connecting each pair of nodes reflects their similarity. Formerly, A graph G(V, E) is a collection of V = {x 1 , . . . , x n } vertices or nodes and E = {e ij } n i,j=1 edges. The n × n adjacency matrix A of a graph G describes the structure of the graph, with each element as a non-negative weight associated with each edge, if two nodes x i and x j are not connected to each other, then A ij = 0. The adjacency matrix A can either be derived using a similarity measure between the data points [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF][START_REF] Iscen | Label propagation for deep semi-supervised learning[END_REF], or be explicitly derived from external data, such as a knowledge graph [START_REF] Wijaya | Pidgin: ontology alignment using web text as interlingua[END_REF], and provided as input. Graph-based tasks can be broadly categorized into four categories [START_REF] Goyal | Graph embedding techniques, applications, and performance: A survey[END_REF]: node classification, link prediction, clustering, and visualization. Graph methods can also be transductive or inductive in nature; transductive methods are only capable of producing labels assignments of the examples seen during training (i.e., the unlabeled nodes of the graph), while inductive methods are more generalizable, and can be transferred and applied to unseen examples. In this section, we will discuss node classification approaches, given that the objective in SSL is to assign labels to the unlabeled examples. Node classification approaches can be broadly grouped into methods which propagate the labels from labeled nodes to unlabeled nodes based on the assumption that nearby nodes tend to have the same labels [START_REF] Azran | The rendezvous algorithm: Multiclass semi-supervised learning with markov random walks[END_REF][START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF][START_REF] Zhou | Learning with local and global consistency[END_REF], and methods which learn node embeddings based on the assumption that nearby nodes should have similar embeddings in vector space and then apply classifiers on the learned embeddings [START_REF] Grover | node2vec: Scalable feature learning for networks[END_REF]. First, we start with some graph construction approaches and then discuss several popular methods for graph-based SSL.

Graph Construction

To apply graph-based SSL, we first need a graph. The graph can either be presented as an input in the form of an adjacency matrix A or can be constructed to reflect the similarity of the nodes. A useful graph should reflect our prior knowledge about the domain and is the practitioner's responsibility to feed a good graph to graph-based SSL algorithms in order to produce valuable outputs (for more details, see Ch3 & 7 [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF]).

In case we have limited domain knowledge about the dataset at hand, Zhu et al. [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF] describes some common ways to create graphs:

• Fully connected graphs. A simple form the graph can take is being fully connected with weighted edges between all pairs of data. With full connectivity, the derivatives of the graph w.r.t., the weights can be computed to update the weights of the edges, but the computational cost, in this case, will be high.

• Sparse graphs. A sparse graph can be constructed so that each node is only connected to a few similar nodes, while the connections to dissimilar nodes are removed. Examples of sparse graphs are kNN graphs where nodes i and j are connected if i is one of k-nearest neighbors [START_REF] Von | A tutorial on spectral clustering[END_REF] of j or vice versa. A possible way to obtain the edge weight A ij between x i and x j is to use a Gaussian kernel [START_REF] Christopher | Pattern recognition and machine learning[END_REF]:

W ij = exp{-x i -x j 2 /2σ 2
} with a hyperparameter σ. Another approach is NN graphs where nodes i and j are connected if the distance d(i, j) ≤ . These graphs can be created using either the raw data or representations extracted from a trained network and updated iteratively (e.g., CNN features [START_REF] Iscen | Label propagation for deep semi-supervised learning[END_REF]).

Label Propagation

The main assumption in label propagation is that the data points in the same manifold are very likely to share the same semantic label [START_REF] Zhou | Learning with local and global consistency[END_REF]. To this end, label propagation propagates labels of the labeled data points to the unlabeled data points according to the data manifold structures and the in-between node similarity.

In label propagation [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF][START_REF] Zhou | Learning with local and global consistency[END_REF][START_REF] Fujiwara | Efficient label propagation[END_REF], the labeling scores are defined as the optimal solution that minimizes the loss function. Let a n × C matrix Ŷ corresponds to the new classification scores for each data point, where each row Ŷi is a probability distribution over C classes, and Y is a n × C matrix containing the labels for the labeled data points, where each row Y i is a one-hot vector if x i is a labeled data point, and a vector of zeros otherwise. The loss function to be minimized for label propagation [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF] is:

L = 1 2 n i,j=1 A ij (ŷ i -ŷj ) 2 = Ŷ T L Ŷ (7.1)
where L = D -A is the graph Laplacian matrix that measures the smoothness of the graph, with D = n j=1 A ij as the degree matrix, this loss function can be viewed as a graph Laplacian regularization which incurs a large penalty when similar nodes with a large weight A ij are predicted to have different labels ŷi = ŷj . By defining a n × n probabilistic transition matrix P = D -1 A, where P ij is the probability of transit from node i to j, and spliting the matrices P , Y and Ŷ into labeled and unlabeled sub-matrices:

P = P ll P lu P ul P uu Y = Y l Y u Ŷ = Ŷl Ŷu (7.2)
the optimal solution for eq. (7.1) is:

Ŷl = Y l Ŷu = (I -P uu ) -1 P ul Y l (7.3)
where I is an identity matrix. The labeling score computation involves the matrix inversion operation, which is computationally heavy for large graphs. As an alternative, Zhu et al. [START_REF] Zhu | Semi-supervised learning using gaussian fields and harmonic functions[END_REF] propose an iterative approach to converge to the same solution:

1. Propagate Ŷ ← P Ŷ .

2. Preserve the labeled data Ŷl = Y l .

3. Repeat from step 1 until convergence.

Another similar label propagation algorithm was proposed by Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF], where, in addition to the contribution a node i receives from its neighbors j, it receives an additional small contribution given by its initial value. In this case, the labels of the labeled nodes might change to better reflect the final labels, which can be helpful if the initial labels are noisy. The loss function in this instance is:

L = 1 2 n i,j=1 A ij Ŷi √ D ii - Ŷi D jj 2 + (1/α -1) n i=1 Ŷi -Y i 2 (7.4)
with a hyperparameter α. The first and second terms in the loss function correspond to the smoothness constraint and the fitting constraint, respectively. The smoothness constraint results in labels that do not change too much between nearby points, while the fitting constraint forces the final labels of the labeled nodes to be similar to their initial value. The optimal solution that minimizes the loss function is:

Ŷ = (I -αS) -1 Y (7.5)
where S = D -1/2 AD -1/2 . Similar the the first algorithm, Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF] propose a less computationally expensive iterative approach:

1. Propagate Ŷ ← αS Ŷ + (1 -α)Y .
2. Repeat step 1 until convergence.

It is worth noting that even though the iterative method is the standard approach for label propagation, it does not output the same labeling results as the the optimal solution.

Graph Embedding

The term graph embedding has been used in the literature in two ways: to represent an entire graph in vector space, or to represent each individual node in vector space [START_REF] Goyal | Graph embedding techniques, applications, and performance: A survey[END_REF]. In this paper, we are interested in learning node embeddings since such a representation can be used for SSL tasks, such as node classification.

The goal of node embedding is to encode the nodes as low dimensional vectors that reflect their positions and the structure of their local neighborhood. These low dimensional embeddings can be viewed as encoding or projecting, nodes into a latent space, where geometric relations in this latent space correspond to interactions (e.g., edges) in the original graph [START_REF] William L Hamilton | Representation learning on graphs: Methods and applications[END_REF][START_REF] Peter D Hoff | Latent space approaches to social network analysis[END_REF]. Factorization-based approaches such as Laplacian Eigenmaps [START_REF] Belkin | Laplacian eigenmaps and spectral techniques for embedding and clustering[END_REF] and Locally Linear Embedding (LLE) [START_REF] Cao | Grarep: Learning graph representations with global structural information[END_REF] are examples of algorithms based on this rationale, but they have scalability issues for large graphs, other more scalable embedding techniques which leverage the sparsity of real-world networks have been proposed. For example, LINE [START_REF] Tang | Line: Large-scale information network embedding[END_REF] and HOPE [START_REF] Ou | Asymmetric transitivity preserving graph embedding[END_REF] attempt to preserve high order proximities (e.g., the edge weights of a given node, and the similarity of edge weights of each pair of nodes). Intuitively, the goal of these methods is simply to learn embeddings for each node such that the inner product between the learned embedding vectors approximates some deterministic measure of graph proximity [START_REF] William L Hamilton | Representation learning on graphs: Methods and applications[END_REF].

Another family of method is random walks introduced by [START_REF] Perozzi | Deepwalk: Online learning of social representations[END_REF] and its variants [START_REF] Grover | node2vec: Scalable feature learning for networks[END_REF][START_REF] Ganguly | Author2vec: Learning author representations by combining content and link information[END_REF][START_REF] Chen | Harp: Hierarchical representation learning for networks[END_REF][START_REF] Abu-El-Haija | Watch your step: Learning node embeddings via graph attention[END_REF]. Instead of using a deterministic measure of graph proximity like factorization-based approaches, these methods optimize the embeddings so that nodes have similar embeddings if they tend to co-occur within short random walks over the graph, making them especially useful when one can either only partially observe the graph or the graph is too large to measure in its entirety. Random walks consist of starting from a randomly samples node x 0 ∈ sample(V ), and then repeatedly sampling an edge to transition to the next node x i+1 = sample(N (x i )), with N (x i ) as the neighboring nodes of x i . The resulting sequence of random walks x 0 → x 1 → x 2 → . . . can then be passed to word2vec algorithm [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] with the objective to embed each node x i within the random walk sequences to be close in the vector space to its neighboring nodes. With a context window of size T , with T usually defined to be in the range T ∈ {2, . . . , 10}, the representation of the anchor node x i is brought closer to the embeddings of its next neighbors {x i-T /2 , . . . , x i , . . . , x i+T /2 }.

Random Walk

Biased random walks Formally, random walk method learn embeddings z i of a given node x i so that:

p T (x j |x i ) ≈ e z i zj x k ∈V e z i z k (7.6)
where p T (x j |x i ) is the probability of visiting x j on a length-T random walk starting at x i . To learn such embeddings, the following loss is optimized:

L = (xi,xj )∈RW -log e z i zj x k ∈V e z i z k (7.7)
where RW is the set of the length-T generated random walks. Evaluating the loss is prohibitively expensive, since assessing the denominator requires a computation over all the nodes of the graph. Thus, different methods use different optimization to approximate the loss in eq. (7.7). For example, DeepWalk [START_REF] Perozzi | Deepwalk: Online learning of social representations[END_REF] uses a hierarchical softmax to compute the denominator, while node2vec approximates eq. (7.7) using negative sampling similar to word2vec. The different methods also differ in the construction of random walk, DeepWalk uses simple unbiased random walks over the graph, while node2vec introduces two random walk hyperparameters, p and q, to smoothly interpolate between walks that are more akin to breadth-first or depth-first search (fig. 17). The hyperparameter p controls the likelihood of the walk immediately revisiting a node, while q controls the likelihood of the walk revisiting its neighborhood [START_REF] William L Hamilton | Representation learning on graphs: Methods and applications[END_REF].

For SSL, the learned embeddings can then be used as inputs to train a classifier over the labeled nodes and then applied over the unlabeled node. Alternatively, a cross-entropy term can be added to the unsupervised loss in eq. (7.7) for SSL based random walks, in order to jointly train a classifier on top of the node embeddings over the labeled nodes. For example, Planetoid [START_REF] Yang | Revisiting semi-supervised learning with graph embeddings[END_REF] introduces a hyperparameter r to control the sampled instances for a given training iteration, alternating between sampling random pairs from a given random walk for the unsupervised loss if r < random, and a couple of nearby labeled nodes with the same label for the supervised loss if r ≥ random. The embeddings are trained using both the supervised and unsupervised loss, while the classifier is only trained with a supervised loss.

Graph Neural Networks

Random walks based method, with their expressivity (i.e., incorporating both local and higher-order neighborhood information) and efficiency (i.e., do not need to consider all node pairs when training), suffer from some limitations, such as the lack of parameter sharing where every node has its own unique embedding, and the inherent transductive nature of these approaches, in which the embeddings are only generated for nodes seen during training. This is especially problematic for evolving graphs, massive graphs that cannot be fully stored in memory, or domains that require generalizing to new graphs after training [START_REF] William L Hamilton | Representation learning on graphs: Methods and applications[END_REF]. To solve these issues, a number of methods use deep neural networks based methods applied to graphs [START_REF] Zhou | Graph neural networks: A review of methods and applications[END_REF][START_REF] Wu | A comprehensive survey on graph neural networks[END_REF][START_REF] Bacciu | A gentle introduction to deep learning for graphs[END_REF]. DNGR [START_REF] Cao | Deep neural networks for learning graph representations[END_REF] and SDNE [START_REF] Wang | Structural deep network embedding[END_REF] propose the first application of deep networks for graphs by using deep autoencoders [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF] in order to compress the information about a node's local neighborhood. A high dimensional representation s i ∈ R |V | of a node x i , which describes the proximity of node x i to all other nodes in the graph is first extracted, and then fed through an autoencoder for dimensionality reduction and trained using a reconstruction loss. After training, the bottleneck low dimensional representation is then used as an embedding for x i . However, these approaches suffer from similar limitations as random walks methods, with inputs of size |V |, which can be extremely costly and even intractable for large graphs, in addition to their transductive nature.

Several recent node embedding approaches aim to solve the main limitations of the random walks and autoencoder based methods by designing functions that rely on a node's local neighborhood (fig. 18), but not necessarily the entire graph. Unlike the previously discussed methods, graph neural networks use the node features, e.g., profile information for a social network or even simple statistics such as node degree [START_REF] Hamilton | Inductive representation learning on large graphs[END_REF] or one-hot vectors [START_REF] Thomas | Variational graph auto-encoders[END_REF], to generate the embeddings. These methods are often called convolutional because they represent a node as a function of its surrounding neighborhood, similar to CNNs [START_REF] William L Hamilton | Representation learning on graphs: Methods and applications[END_REF]. The training procedure starts by initializing the first hidden states h 0 i using the nodes features x i : h 0 i ← x i , ∀x i ∈ V . For K training iterations, at each step, the hidden stated are updated by aggregating the hidden states of the neighboring nodes, with an AGGREGATE, COMBINE, a non-linearity σ, and a NORMALIZE functions as follows [START_REF] Hamilton | Inductive representation learning on large graphs[END_REF]:

For k = 1 . . . K:

For x i ∈ V : 1. h ← AGGREGATE k ({h k-1 j , ∀x j ∈ N (x i )}) 2. h k i ← σ(W k • COMBINE(h k-1 i , h )) 3. h k i ← NORMALIZE(h k i )
and at the end, the embeddings z i of node x i are the final hidden states: z i ← h K i . The aggregation function and the set of trainable parameters W k , ∀k ∈ [1, K] specify how to aggregate the local neighborhood information. The different approaches such as GCN [START_REF] Scarselli | The graph neural network model[END_REF][START_REF] Tang | Line: Large-scale information network embedding[END_REF][START_REF] Kearnes | Molecular graph convolutions: moving beyond fingerprints[END_REF][START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF], GraphSAGE [START_REF] Hamilton | Inductive representation learning on large graphs[END_REF] and GAT [START_REF] Veličković | Graph attention networks[END_REF] follow the same procedure but differ primarily in how the aggregation, the combination and the normalization are performed. For example, GraphSAGE uses concatenation as a combination function and experiment with various general aggregation functions, i.e., the element-wise mean, max-pooling, and LSTMs, while GCN uses a weighted sum as a combination function and element-wise mean as an aggregate. The weight are then trained using an unsupervised loss similar to random walks based methods, and for SSL [START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF], a classifier is trained on top of the node embeddings (i.e., the final hidden state) to predict the class labels for the labeled nodes, which can then applied on the unlabeled nodes for node classification.

8 Self-Supervision for SSL Self-supervised learning [START_REF] Weng | Self-supervised representation learning[END_REF][START_REF] Jing | Self-supervised visual feature learning with deep neural networks: A survey[END_REF] is a form of unsupervised learning, where the model is trained using a standard supervised loss, but on a pretext task where the supervision comes from the data itself. The objective, in this case, is not to maximize final performance on the pretext task, but rather to learn rich and transferable features for downstream tasks. A variety of pretext tasks were proposed, where the model is first trained on one or multiple tasks with unlabeled examples, the resulting model is either used for generating representations for the raw data, which are utilized for training a shallow classifier on D l , or directly fine-tuned for a downstream task with labeled images. Examples of such pretext tasks for computer vision are:

• Exemplar-CNN [START_REF] Dosovitskiy | Discriminative unsupervised feature learning with exemplar convolutional neural networks[END_REF]. For a given image, a set of N patches are generated using different transformations, all these patches are then considered as a separate class, and the model is trained to predict the correct class for a given input patch.

• Rotation [START_REF] Gidaris | Unsupervised representation learning by predicting image rotations[END_REF]. A given rotation out of four possible rotations of multiple of 90 is applied to the input image, and the model is trained to predict the correct rotation that was applied.

• Patches [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF]. A first patch is randomly extracted from the input image, this patch is considered as the center, and eight different neighboring and non-overlapping patches are extracted with small jitters at the eight neighboring locations, the model is then trained to predict the position of one of the second patches with regard to the first one. Other versions of this pretext task were proposed, such as jigsaw puzzle [START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF] where a random permutation of the nine patches are fed into the model, and the objective is to predict the correct permutation that was applied to get the correct ordering of the patches.

• Colorization [START_REF] Zhang | Colorful image colorization[END_REF]. The input image is first transformed from RGB to Lab color space, an input with only the luminance information contained within the L component or the color information with ab components is fed into the model, and the objective is to predict the rest of the information, i.e., either the luminance or the coloring of the image. The task can either be considered as a regression problem or a classification problem by quantizing the Lab color space.

• Contrastive Predictive Coding [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF]. Using a contrastive loss based on Noise Contrastive Estimation [START_REF] Gutmann | Noise-contrastive estimation: A new estimation principle for unnormalized statistical models[END_REF] and its recent versions such as Momentum Contrast [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] and SimCLR [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF], the model is trained to differentiate between positive and negative samples, the positives can be a given input image and its transformed versions, or a given patch and its neighboring patches, while the negatives are randomly sampled images or patches.

Such pretext tasks can easily be utilized for SSL, where the model is trained on the whole dataset on the pretext task with self-supervision, and then adapted to the labeled set D l using the standard cross-entropy loss, either simultaneously as demonstrated by [START_REF] Zhai | S4l: Self-supervised semi-supervised learning[END_REF][START_REF] Berthelot | Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring[END_REF] with rotation as the pretext task, or iteratively, by first training the model using self-supervision and then fine tuning it on D l as demonstrated by [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Chen | Big self-supervised models are strong semi-supervised learners[END_REF] using contrastive learning.

Conclusion

In this paper, we introduced semi-supervised learning, with its main approaches and assumptions, with SSL techniques within deep learning framework. Specifically, this review covered four broad categories of approaches for SSL: consistency regularization, generative models, graph-based methods, and holistic approaches. With the growing research interest in data-efficient deep learning algorithms, it is foreseeable that deep SSL methods could approach the performance of fully supervised methods, and have board applications integrated into different systems and learning paradigms.

Figure 1 :

 1 Figure 1: SSL toy example. The decision boundaries obtained on two-moons dataset, with a supervised and different SSL approaches using 6 labeled examples, 3 for each class, and the rest of the points as unlabeled data.

Figure 4 :

 4 Figure 4: Loss computation for Temporal Ensembling.The MSE between the current prediction and the aggregated target is computed for the unsupervised loss, and if the input is a labeled example, we add the supervised loss to the weighted unsupervised loss.

Figure 5 :

 5 Figure 5: Mean Teacher. The teacher model, which is an EMA of the student model, is responsible for generating the targets for consistency training. The student model is then trained to minimize the supervised loss over labeled examples and the consistency loss over unlabeled examples. At each training iteration, both models are evaluated with an injected noise (η, η ), and the weights of the teacher model are updated using the current student model to incorporate the learned information at a faster pace.

Figure 6 :

 6 Figure 6: SWA and fast-SWA. Left and Center. Cyclical cosine learning rate schedule used at the end of training for SWA and fast-SWA with different averaging strategies. Right. 2d illustration of the impact of SWA and fast-SWA averaging strategies on the final weights. Based on [5].

Figure 7 :

 7 Figure 7: Virtual Adversarial Examples. Examples of the perturbed ImagetNet images for different values of the scaling hyperparameter .

Figure 8 :Figure 9 :

 89 Figure 8: EAdD and CAdD. EAdD drops activation individually regardless of the spatial correlation, while CAdD drops entire feature maps, making it more suitable for convolutional layers. Image Source: [95].

  , as perturbations for consistency training based SSL. Similar to supervised learning, advanced data augmentation methods can also provide extra advantages over simple augmentations and random noise injection in consistency training, given that; (1) it generates realistic augmented examples, making it safe to encourage the consistency between predictions on the original and augmented examples, (2) it can generate a diverse set of examples improving the sample efficiency, and (3) it is capable of providing the missing inductive biases for different tasks.

Figure 10 :

 10 Figure 10: UDA. The training procedure consists of computing the supervised loss for the labeled examples and the consistency loss between the two outputs of the augmented and clean input.

Figure 11 :

 11 Figure 11: The MPL training procedure. At each training iteration, the teacher model is trained along with a student model to set the student's target distributions and adapt to the student's learning state. Image Source: [120].

Figure 12 :

 12 Figure 12: Cross-view Training. An example of auxiliary student prediction modules. Each student sees a restricted view of the input.For instance, the forward prediction module does not see any context to the right of the current token when predicting that tokens label. Image Source:[START_REF] Clark | Semi-supervised sequence modeling with cross-view training[END_REF]

Figure 13 :

 13 Figure 13: MixMatch. The procedure of label guessing process used in MixMatch, taking as input a batch of unlabeled examples, and outputting a batch of K augmented version of each input, with a corresponding sharpened proxy labels. Image Source: [120].

Figure 14 :

 14 Figure 14: ReMixMatch. Left. Distribution alignment adjusts the guessed labels distributions to match the ground-truth class distribution divided by the average model predictions on D u . Right. Augmentation anchoring uses the prediction obtained using a weakly augmented image as targets for a strongly augmented version of the same image. Image Source: [12].

  First, a new unlabeled batch Û of examples is created by rotating all of the examples with an angle r ∼ {0, 90, 180, 270}. The rotated examples are then used to compute a self-supervised loss, where the classification layer on top of the model predicts the correct applied rotation, in addition to the cross-entropy loss over the rotated examples:

Figure 16 :

 16 Figure 16: GAN framework. During training, the discriminator D alternates between receiving real samples from the data distribution p(x), with the goal of correctly classifying them as real, i.e., D(x) = 1, and generated samples G(z) with the aim of correctly classifying them as fake, i.e., D(G(z)) = 0, while competing with the generator, trying to generate real-looking samples to fool the discriminator, i.e., D(G(z)) = 1.

E

  (x,y)∼p(x,y) [log D(x, y)] + αE (x,y)∼pc(x,y) [log(1 -D(x, y))] + (1 -α)E (x,y)∼pg(x,y) [log(1 -D(G(y, z), y))] (6.10)

Figure 17 :

 17 Figure 17: Random Walks. Left. An example of a random walk of length 4 starting from node x 1 : x 1 → x 3 → x 5 → x 7 → x 6 Right. Breadth first search (BFS) and depth first search (DFS) strategies from node x 3 .

Figure 18 :

 18 Figure 18: Context Aggregation. An example of a three-step context aggregation. The context of x 1 at k = 2 depends not only on its neighboring node x 2 , but also the neighbors of x 2 due to the first aggregation step.

Acknowledgements

Y. Ouali is supported by Randstad corporate research in collaboration with Université Paris-Saclay, Cen-traleSupélec, MICS. We thank Victor Bouvier for his helpful feedback on an earlier version.