Kushal Lakhotia

Hsu Eugene Kharitonov

Wei-Ning Hsu

Yossi Adi

Adam Polyak

Benjamin Bolte

Tu-Anh Nguyen

Jade Copet

Alexei Baevski

Abdelrahman Mohamed

Emmanuel Dupoux

On Generative Spoken Language Modeling from Raw Audio

Keywords:

We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo-text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder-dependent way, and that some combinations approach text-based systems.

Introduction

An open question for AI research is creating systems that learns from natural interactions as infants learn their first language(s): from raw uncurated data, and without access to text or expert labels [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner[END_REF]. Natural Language Processing (NLP) systems are currently far from this requirement. Even though great progress has been made in reducing or eliminating the need for expert labels through self-supervised training objectives [START_REF] Tom | Language models are few-shot learners[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]Liu et al., 2019b;[START_REF] Dong | Unified language model pre-training for natural language understanding and generation[END_REF][START_REF] Lewis | BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[END_REF], the basic units on which these systems are trained are still textual. Yet, young children learn to speak several years before they can read and write, providing a proof of principle that language can be learned without any text. Being able to achieve 'textless NLP' would be beneficial for the majority of the world's languages which do not have large textual resources or even a widely used standardized orthography (Swiss German, dialectal Arabic, Igbo, etc.), and which, despite being used by millions of users, have little chance of being served by current text-based technology. It would also be useful for 'high-resource' languages, where the oral and written forms often mismatch in terms of lexicon and syntax, and where some linguistically relevant signals carried by prosody and intonation are basically absent from text. While text is still the dominant form of language on the web, a growing amount of audio resources like podcasts, local radios, social audio apps, on-line video games provide the necessary input data to push NLP to an audio-based future and thereby expand the inclusiveness and expressivity of AI systems.

Is it possible to build an entire dialogue system from audio inputs only? This is a difficult challenge, but breakthroughs in unsupervised representation learning may address part of it. Unsupervised learning techniques applied to speech were shown to learn continuous or discrete representations that capture speaker invariant phonetic content [START_REF] Curran Associates | The Zero Resource Speech Challenge 2015: Proposed approaches and results[END_REF][START_REF] Dunbar | The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units[END_REF], despite themselves not being phonemic [START_REF] Schatz | Early phonetic learning without phonetic categories: Insights from large-scale simulations on realistic input[END_REF]. Recent developments in selfsupervised learning have shown impressive results as a pretraining technique [START_REF] Van Den Oord | Neural discrete representation learning[END_REF][START_REF] Chung | An unsupervised autoregressive model for speech representation learning[END_REF][START_REF] Hsu | HuBERT: How much can a bad teacher benefit ASR pre-training?[END_REF], to the extent that Automatic Speech Recognition (ASR) on par with the state of the art from two years back can be built with 5000 times less labelled speech (Baevski et al., 2020b), or even no with no labelled speech at all [START_REF] Baevski | Unsupervised speech recognition[END_REF]. Of course, ASR still assumes access to text to learn a lan-Table 1: Tasks and metrics proposed to evaluate encoding/generation quality of models at the acoustic or language levels. Bold fonts highlights the main metric used for each category (Section 3 for details).

guage model (LM) and the mapping to the audio units. Here, we study the case where the LM is directly trained from the audio units without any recourse to text.

The high level idea (see Figure 1) is that automatically discovered discrete units can be used to encode speech into "pseudo-text" (speech-tounit, S2u), which is used in turn to train a generative language model (unit-based language model, uLM) and to train a speech synthesizer (unit-tospeech, u2S). This enables learning an LM from scratch without text, and use it to generate speech conditionally or unconditionally, essentially replicating what toddlers achieve before learning to read. Early studies using discrete codes learned from an autoencoder show the feasibility of such an approach, but remain at a level of a demo [START_REF] Van Den Oord | Neural discrete representation learning[END_REF].

In this paper, we address one major conceptual stumbling block which has, thus far, prevented such early studies from having the transformative impact they could have in language technology: model evaluation. We contend that it will be impossible to make progress in this area beyond demos unless proper evaluation methods enabling system comparison are etablished.

Evaluation for speech generation is difficult due to the continuous, variable and multi-level nature of the speech waveform, and the necessity both to capture fine grained acoustic details to generate intelligible audio and to abstract away from them to learn higher level language concepts. Text-based models do not have this problem, since the input is already expressed in terms of mid-level discrete units (characters or words), and are typically evaluated with unsupervised metrics close to the learning objectives like perplexity or log likelihood. Here, such an approach is not directly applicable even if we rely on discrete pseudo-text units, since such metrics would depend in an unknown fashion on their granularity (number, duration and distribution), making the comparison of models that use different units infeasible.

Conceptually, generative spoken language models can be evaluated at two levels, the acoustic and the language levels, and through two modes of operation, encoding and generation, resulting in 2x2 tasks (see Table &Figure 1). Acoustic Unit Discovery (encoding at the acoustic level) consists in representing speech in terms of discrete units discarding non-linguistic factors like speaker and noise. Spoken Language Modeling (encoding at the language level) consists in learning the probabilities of language patterns. Speech Resynthesis (generation for acoustic modeling) consists in generating audio from given acoustic units. This boils down to repeating in a voice of choice an input linguistic content encoded with speech units. Speech Generation (generation for language modeling) consists in generating novel and natural speech (conditioned on some prompt or not). Compared to standard text generation, a critical and novel component of the audio variant is clearly the discovery of units since it conditions all the other components. This is why we devote our analyses of model architectures to the unit-to-speech component specifically, and leave it for further work to evaluate how the downstream components can also be optimized for spoken language generation.

The major contributions of this paper are as follows : (1) we introduce two novel evaluation metrics for the generation mode of spoken language modeling at the acoustic and language levels respectively. Our key insight is to use a generic pretrained ASR system to establish model-independent assessments of the intelligibility (acoustic level) and meaningfulness (language level) of the produced outputs. The ASR system converts the generated waveform back to text, enabling us to adapt standard text-based metrics for these two levels. (2) we validate these metrics through comparison with human evaluation. We show a high degree of concordance between human and machine evaluations of intelligibility and meaningfulness of generated audio. (3) we show that these metrics can be predicted by simpler ones geared to evaluate the encoding mode of the spoken LM. Zero-shot metrics borrowed from previous studies in the Zero Resource Speech Challenges [START_REF] Curran Associates | The Zero Resource Speech Challenge 2015: Proposed approaches and results[END_REF][START_REF] Tu | The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling[END_REF] correlate well with their generative counterpart, offering an easier proxy to rapidly iterate on model selection. (4) we systematically study the effect of the type of encoding units by factorially crossing three recent speech-to-unit encoders, CPC, Wave2vec 2.0 and HuBERT, with three codebook sizes for the discrete units, 50, 100, 200. We keep constant the rest of the system built from out-ofthe-box components (standard Transformer for the uLM, Tacotron 2 for u2S). We show that both the encoder type and the number of units matter, and that they matter differently depending on the evaluation task. ( 5) we open source our evaluation tools and models to help reproducibility and comparability with future work.

In Section 3, we introduce the ASR, zero-shot and human evaluation metrics, in Section 4 we present the models, in Section 5, we analyze the results and discuss them in Section 6.

Related work

Unsupervised speech representation learning aims to distill features useful for downstream tasks, such as phone discrimination [START_REF] Kharitonov | Data augmenting contrastive learning of speech representations in the time domain[END_REF][START_REF] Schneider | wav2vec: Unsupervised pre-training for speech recognition[END_REF] and semantic prediction [START_REF] Lai | Semisupervised spoken language understanding via self-supervised speech and language model pretraining[END_REF][START_REF] Wu | Self-supervised representations improve end-to-end speech translation[END_REF], by constructing pretext tasks that can exploit large quantities of unlabeled speech. Pretext tasks in the literature can be roughly divided into two categories: reconstruction and prediction. Recon-struction is often implemented in the form of autoencoding (Hsu et al., 2017a), where speech is first encoded into a low-dimensional space, and then decoded back to speech. Various constraints can be imposed on the encoded space, such as temporal smoothness [START_REF] Ebbers | Hidden Markov Model variational autoencoder for acoustic unit discovery[END_REF][START_REF] Glarner | Full Bayesian Hidden Markov Model variational autoencoder for acoustic unit discovery[END_REF][START_REF] Khurana | A factorial deep Markov Model for unsupervised disentangled representation learning from speech[END_REF][START_REF] Khurana | A convolutional deep Markov Model for unsupervised speech representation learning[END_REF], discreteness [START_REF] Ondel | Variational inference for acoustic unit discovery[END_REF][START_REF] Van Den Oord | Neural discrete representation learning[END_REF], and presence of hierarchy [START_REF] Lee | A nonparametric Bayesian approach to acoustic model discovery[END_REF]Hsu et al., 2017b). Prediction-based approaches task a model with predicting information of unseen speech based on its context. Examples of information include spectrograms [START_REF] Chung | An unsupervised autoregressive model for speech representation learning[END_REF][START_REF] Wang | Unsupervised pre-training of bidirectional speech encoders via masked reconstruction[END_REF][START_REF] Chi | Audio ALBERT: A lite BERT for self-supervised learning of audio representation[END_REF][START_REF] Liu | Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders[END_REF][START_REF] Chung | Improved speech representations with multi-target autoregressive predictive coding[END_REF][START_REF] Liu | Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders[END_REF]Ling et al., 2020;Ling and Liu, 2020), cluster indices [START_REF] Baevski | Effectiveness of selfsupervised pre-training for speech recognition[END_REF][START_REF] Hsu | HuBERT: How much can a bad teacher benefit ASR pre-training?[END_REF], derived signal processing features [START_REF] Pascual | Learning problem-agnostic speech representations from multiple self-supervised tasks[END_REF][START_REF] Ravanelli | Multi-task self-supervised learning for robust speech recognition[END_REF], and binary labels of whether a candidate is the target unseen spectrogram [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Schneider | wav2vec: Unsupervised pre-training for speech recognition[END_REF]Baevski et al., 2020a;[START_REF] Kharitonov | Data augmenting contrastive learning of speech representations in the time domain[END_REF]Baevski et al., 2020b).

Speech resynthesis. Recent advancements in neural vocoders enabled generating natural sounding speech and music [START_REF] Van Den Oord | WaveNet: A generative model for raw audio[END_REF][START_REF] Kumar | Mel-GAN: Generative adversarial networks for conditional waveform synthesis[END_REF][START_REF] Kong | HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis[END_REF]. These are often conditioned on the log mel-spectrogram for the generation process. Learning low bitrate speech representations in an unsupervised manner, has attracted attention from both the machine learning and the speech communities (Liu et al., 2019a;[START_REF] Feng | Combining adversarial training and disentangled speech representation for robust zeroresource subword modeling[END_REF][START_REF] Nayak | Virtual phone discovery for speech synthesis without text[END_REF][START_REF] Tjandra | VQVAE unsupervised unit discovery and multi-scale Code2Spec inverter for Zerospeech Challenge[END_REF][START_REF] Schneider | wav2vec: Unsupervised pre-training for speech recognition[END_REF]Baevski et al., 2020a;[START_REF] Chen | Unsupervised acoustic unit representation learning for voice conversion using WaveNet auto-encoders[END_REF][START_REF] Morita | Exploring TTS without T using biologically/psychologically motivated neural network modules (ZeroSpeech 2020)[END_REF][START_REF] Patrick Lumban Tobing | Cyclic spectral modeling for unsupervised unit discovery into voice conversion with excitation and waveform modeling[END_REF]. These representations can later be used for generation without text, which is particularly important for lowresource languages [START_REF] Dunbar | The Zero Resource Speech Challenge 2019: TTS without T[END_REF][START_REF] Dunbar | The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units[END_REF]. [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] proposed a Vector-Quantized Variational Auto-Encoder (VQ-VAE) model to learn discrete speech units, which will be later used for speech synthesis using a WaveNet model. [START_REF] Eloff | Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks[END_REF] suggested a VQ-VAE model followed by a FFTNet vocoder model [START_REF] Jin | FFTNet: A real-time speaker-dependent neural vocoder[END_REF]. [START_REF] Tjandra | Transformer VQ-VAE for unsupervised unit discovery and speech synthesis: ZeroSpeech 2020 Challenge[END_REF] suggested to use transformer [START_REF] Vaswani | Attention is all you need[END_REF] together with a VQ-VAE model for unsupervised unit discovery, and van Niekerk et al. (2020) combines vector quantization together with contrastive predictive coding for acoustic unit discovery. Another line of work use representations from an ASR acoustic model that are combined with identity and prosodic information for voice conversion (Polyak et al., 2020b(Polyak et al., ,a, 2021b)). In terms of evaluation, the Zero-Resource challenge [START_REF] Dunbar | The Zero Resource Speech Challenge 2019: TTS without T[END_REF][START_REF] Dunbar | The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units[END_REF][START_REF] Tu | The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling[END_REF] used bitrate together with human evaluation. In this paper we additionally introduce an ASR based evaluation metric.

Evaluation Methods

We present two sets of automatic evaluation metrics; the first ones assess the output of generative speech models (ASR metrics, Section 3.1); the second ones, the encoded representations (zeroshot probe metrics, Section 3.2). Finally, we present the human evaluations (Section 3.3).

Generation: ASR metrics

We present our new evaluation metrics for generation tasks. The first task, speech resynthesis, involves S2u which encodes input speech into units and u2S which decodes it back to speech. In this task, we wish to evaluate intelligibility of the resulting speech. The second task, speech generation, involves the full S2u→uLM→u2S pipeline, and we wish to evaluate meaningfulness of the generated speech. Our overall idea is to use ASR to convert the generated speech back to text and then use text-based metrics. Speech resynthesis intelligibility: ASR-PER. The ideal metric for intelligibility would be to use humans to transcribe the resynthesized speech and compare the text to the original input. An automatic proxy can be obtained by using a state-ofthe-art ASR system pretrained on a large corpus of real speech. 2 Our main metric is Phone Error Rate (PER), which only uses an acoustic-model ASR, without fusing with an additional language model [START_REF] Chorowski | Towards better decoding and language model integration in sequence to sequence models[END_REF]. In preliminary experiments we also experimented with a full ASR with an LM and computed Word Error Rate (WER) and Character Error Rate (CER) to give partial credit. The latter is probably closer to humans intelligibility metrics, as humans cannot turn off their lexicon or language model. We also computed such metrics by training a fitted ASR model for each resynthesis model on a specific training corpus (see Supplementary Section S2.1). The logic of this last test is that it provides a more direct measure of the information lost in the 2 We use a BASE wav2vec 2.0 phoneme detection model trained on LibriSpeech-960h with CTC loss from scratch.

S2u→u2S pipeline, because it could adapt to systematic errors introduced by the u2S model. Since the scores between these different approaches correlated highly, we only report here the results on the PER for a pretrained ASR model which is the simplest to deploy. Speech generation quality and diversity: AUC on Perplexity and VERT. Text generation evaluation typically involves two axes: the quality of the generated text (with automatic metrics like mean perplexity or negative log likelihood computed on a reference large language model) and the diversity (with metrics like self-BLEU3 , [START_REF] Zhu | Texygen: A benchmarking platform for text generation models[END_REF]. Typically, there is a trade-off between these two dimensions based on the temperature hyperparameter used for sampling from the language model, whereby at low temperature, the system outputs good sentences but not varied, and at high temperatures, it outputs varied sentences, but not very good. This results in model comparison being either based on 2D plots with lines representing the trade-off between quality and diversity, or on aggregate metrics like the area under the curve. Preliminary explorations (see Appendix Section 7.2) with our models revealed two problems preventing a straightforward application of such a scoring strategy.

First, we found that for some models, at a low enough temperature, self-BLEU score stopped increasing, but the systems started to repeat more and more words within a sentence (e.g., "the property the property the property"). We therefore introduce a new metric, auto-BLEU, that measures within-sentence diversity. For a single utterance u, auto-BLEU is calculated as the ratio of k-grams s ∈ N G k (u) that are repeated at least once:

auto-BLEU(u, k) = s 1 [s ∈ (N G k (u)\s)] |N G k (n)| (1)
As with BLEU score, to get n-gram auto-BLEU we calculate the geometric mean of auto-BLEU(u, k) obtained for k ∈ [1, n] and average over the set of generated utterances. By calculating the geometric mean of self-and auto-BLEU, we obtain an aggregate metric which we call VERT (for diVERsiTy). We used a bigram version of self-and auto-BLEU. Second, we found that critical temperatures for which the output was reasonable were not constant across models. This makes sense, because temperature controls the probability of sampling individual units, and the probabilistic distribution and duration of these units depend on the models. Here, we chose to use the oracle text as an anchor to compute reference temperatures, i.e., the temperatures at which the perplexity or the VERT score reach the values of the oracle text.

This gives us boundary conditions at which we can compare (the perplexity at oracle diversity and the diversity at oracle perplexity), as well as a method to compute the area under curve (AUC) between these two boundaries (See Figure 2). As AUC decreases, the system gets closer to the oracle point. Thus with AUC, lower is better.

To calculate perplexity of the generated utterances, we use a pre-trained ASR4 to convert speech to text, and an off-the-shelf Transformer model trained on the English NewsCrawl dataset.5

Encoding: Zero-shot probe metrics

The purpose of the encoding metrics is to evaluate the quality of the learned representations at each linguistic level along the pipeline linking the S2u and the uLM. They are inspired by human psycholinguistics and can be be thought of as unit tests providing interpretation and diagnosis. We entirely draw on evaluations from the Zero Resource challenge series [START_REF] Curran Associates | The Zero Resource Speech Challenge 2015: Proposed approaches and results[END_REF][START_REF] Dunbar | The Zero Resource Speech Challenge 2019: TTS without T[END_REF][START_REF] Tu | The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling[END_REF] 6 for comparability with published work and refer to these challenges for details. These metrics are "zeroshot" because they do not require training any classifier, and are either based on distances over embeddings, or on computing probabilities over entire utterances. When they have hyperparameters, these are selected using a validation set.

For acoustic-level evaluation, we use the between-speaker ABX score to quantify how well-separated phonetic categories are. Briefly, it consists in estimating the probability that two tokens of the same category A (x and a) are closer to one another than a token of A (x) and of B (b). The categories are triphones that only differ in the middle phoneme (like bit and bet) and the score is averaged over all possible such pairs. For the across-speaker ABX, a and b are spoken by the same speaker and x by a different one, requiring feature invariance over a speaker change. We also include the bitrate which has been used in the TTS-without-T challenges [START_REF] Dunbar | The Zero Resource Speech Challenge 2019: TTS without T[END_REF] to quantify the efficiency of the discrete units used to resynthetize speech. It is simply the entropy of the sequence of units divided by the total duration.

For language-level evaluation, we use spot-theword accuracy from the Zero Resource 2021 Benchmark [START_REF] Tu | The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling[END_REF]. It consists in detecting the real word from a pair of short utterances like 'brick' vs 'blick', matched for unigram and bigram phoneme frequency to ensure that low-level cues do not make the task trivial. This task can be done by computing the probability (or pseudo-probability) of the utterances from the uLM. The test set (sWUGGY) consists of 5,000 word-pseudoword pairs generated by the Google TTS API, filtered for the word being present in the LibriSpeech 960h training set [START_REF] Panayotov | LibriSpeech: an ASR corpus based on public domain audio books[END_REF]. The ZR21 benchmark also uses higher level metrics, notably, syntactic (based on the sBLIMP dataset), which we did not use because the baselines were too close to chance.

Human evaluation metrics

As above, we asked humans to evaluate two aspects of speech generation: intelligibility and meaningfulness. Intelligibility was assessed using two metrics: i) Mean Opinion Scores (MOS) in which raters were asked to evaluate subjectively how intelligible a given audio sample is; ii) Character Error Rate (CER) computed from written transcriptions providing an objective intelligibility test. As for meaningfulness, we set up a meaningfulness-MOS (MMOS) in which raters were asked to evaluate how natural (considering both grammar and meaning) a given sample is. For both subjective tests raters evaluate the samples on a scale of 1-5 with an increment of 1.

For the MMOS, we had to select a temperature to sample from. Preliminary experiments showed that humans preferred lower temperatures (yielding also less diverse outputs, see Supplementary Section S2.2). Here, we settled on selecting the temperature on a model-by-model basis by constructing a continuation task: we take the 1,000 shortest utterances from LibriSpeech testclean that are at least 6 seconds long, and use the first 3 seconds as prompts for the uLM (after transcribing them into pseudo-texts). For each prompt, we generated 10 candidate continuations of the same length (in seconds) as the utterance which we took the prompt from. We varied temperature (0.3, 0.4, ..., 1.4, 1.5, 1.7, 1.9, 2.1, 2.3, 2.5, 3.0), and selected the one yielding the maximal BLEU-2 score with the reference sentence (after ASR). These temperatures were typically between the two boundary temperatures described above.

We evaluated 100 samples from each of the evaluated methods while we enforced at least 15 raters for each sample. The CrowdMOS package [START_REF] Ribeiro | CROWDMOS: An approach for crowdsourcing mean opinion score studies[END_REF] was used for all subjective experiments using the recommended recipes for detecting and discarding inaccurate scores. The recordings for the naturalness test were generated by the LM unconditionally and conditionally from a 3 seconds prompt. Participants were recruited using a crowd-sourcing platform.

Proposed Systems

Here, we present our S2u (Section 4.1), uLM (Section 4.2) and u2S (Section 4.3) components.

Speech-to-unit Models

We selected 3 recent state-of-the-art unsupervised Encoders, which we used 'out of the box': we did not retrain them nor change their hyperparameters. We also included a log Mel filter-bank baseline (80 filters, computed every 10ms). We then discretized the embeddings using k-means. We only give a high level description of these models, and refer to the original publications for details. CPC. Contrastive Predictive Coding (van den [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] as applied to speech consists of two components: an encoder and a predictor. The encoder produces an embedding z from speech input. The predictor predicts the future states of the encoder based on the past, and the system is trained with a contrastive loss. We use the CPC model from [START_REF] Rivière | Towards unsupervised learning of speech features in the wild[END_REF], which was trained on a "clean" 6k hour sub-sample of the LibriLight dataset [START_REF] Kahn | Libri-light: A benchmark for ASR with limited or no supervision[END_REF][START_REF] Rivière | Towards unsupervised learning of speech features in the wild[END_REF]. We extract a representation from an intermediate layer of the predictor, which provides a 256-dimensional embedding (one per 10ms), as in the original paper. wav2vec 2.0. Similar to CPC, this model uses an encoder and a predictor, which is trained contrastively to distinguish positive and negative samples from discretized and masked segments of the encoder's output. We use the LARGE variant of pretrained wav2vec 2.0 (Baevski et al., 2020b) trained on 60k hours of LibriLight dataset [START_REF] Kahn | Libri-light: A benchmark for ASR with limited or no supervision[END_REF]. This model encodes raw audio into frames of 1024-dimensional vectors (one per 20ms). To choose the best layer, we extracted frozen representations of the 10 hour LibriLight subset from every layer of the model and trained a linear classifier with the CTC loss to predict the phonetic version of the text labels. Layer 14 obtained the lowest PER on LS dev-other (a similar approach was done in [START_REF] Baevski | Unsupervised speech recognition[END_REF] which in this case selected Layer 15). HuBERT. Unlike CPC and wav2vec 2.0 that use a contrastive loss, HuBERT is trained with a masked prediction task similar to BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] but with masked continuous audio signals as inputs. The targets are obtained through unsupervised clustering of raw speech features or learned features from earlier iterations, motivated by DeepCluster [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF]. We use the BASE 12 transformer-layer model trained for two iterations [START_REF] Hsu | HuBERT: How much can a bad teacher benefit ASR pre-training?[END_REF] on 960 hours of Lib-riSpeech [START_REF] Panayotov | LibriSpeech: an ASR corpus based on public domain audio books[END_REF]. This model encodes raw audio into frames of 768-dimensional vectors (one per 20ms) at each layer and we extract those from the 6 th layer as in the original paper. LogMel. As a baseline, we consider a Log Mel Filterbank encoder using 80 frequency bands. Quantization. We use k-means to convert continuous frame representations into discrete representation by training on LibriSpeech clean-100h [START_REF] Panayotov | LibriSpeech: an ASR corpus based on public domain audio books[END_REF]. We experiment with codebooks that have 50, 100, and 200 units.

unit-Language Model

We use the Transformer model as implemented in fairseq [START_REF] Ott | Fairseq: A fast, extensible toolkit for sequence modeling[END_REF]. We use the trans-former_lm_big architecture: it has 12 layers, 16 attention heads, embedding size of 1024, FFN size Figure 3: Overall results with automatic and human metrics. The results are presented in terms of bitrate for 4 encoders (LogMel, CPC, HuBERT and wav2vec 2.0) varying in number of units (50,100,200). For definition of the tasks and metrics, see Table 1 andFigure 1. Negative human opinion scores are shown for ease of comparison with automatic metrics (lower is better). The generation metrics have been averaged across LS and LJ (PER and MOS; resynthesis task) and across prompted and unprompted conditions (AUC and MMOS; speech generation task). The Log Mel Fbank based systems were not evaluated by humans in the speech generation task.

of 4096, and dropout probability of 0.1, and we train it as a causal LM on sequences of pseudotext units. Each sample contains up to 3,072 units. We use sampling with temperature for generation.

All language models are trained on "clean" 6k hours sub-sample of LibriLight used in [START_REF] Rivière | Towards unsupervised learning of speech features in the wild[END_REF], transcribed with corresponding discrete units. In preliminary experiments, we found that removing sequential repetitions of units improves performance, hence we apply it universally. 7 We hypothesise that this simple modification allows to use Transformer's limited attention span more efficiently as in [START_REF] Hsu | Text-free image-tospeech synthesis using learned segmental units[END_REF].

unit-To-Speech Model

We adapt the Tacotron-2 model [START_REF] Shen | Natural TTS synthesis by conditioning WaveNet on MEL spectrogram predictions[END_REF] such that it takes pseudo-text units as input and outputs a log Mel spectrogram. To enable the model to synthesize arbitrary unit sequences, including those representing incomplete sentences, we introduce two modifications. First, we append a special "end-of-input" (EOI) token to the input sequence, hinting the decoder to predict the "endof-output" token when attending to this new token. However, this modification alone may not be sufficient, as the decoder could still learn to ignore the EOI token and correlate end-of-output prediction with the learned discrete token that represents silence as most of the speech contains trailing silence. To address this, we train the model using random chunks of aligned unit sequence and spectrogram, and append the EOI token to unit sequence chunks, such that the audio does not always end with silence. We implement chunking in the curriculum learning fashion, where the chunk size gradually grows (starting with 50 frames with an increment of 5 per epoch) to increase the difficulty of the task. For waveform generation, we use the pre-trained flow-based neural vocoder WaveGlow [START_REF] Prenger | Waveglow: A flow-based generative network for speech synthesis[END_REF]. This model outputs the time-domain signal given the log Mel spectrogram as input. All u2S models were trained on LJ Speech (LJ) [START_REF] Ito | The lj speech dataset[END_REF].

Results

In Figure 3, we report the overall results of our models and our LogMel baseline as a function of the number of quantized units on our main automated and human metrics. More detailed results follow in the following sections, including two character-based toplines: one uses the oracle transcripts for training the LM, the other uses transcripts produced by the pre-trained ASR model.

Results on the resynthesis task

Overall resynthesis results are shown in the bottom middle and right cells of Figure 3 for our main automatic (PER) and human scores (MOS), respectively, averaged across the LS and LJ evaluation sets. We observe that across all models, increasing the number of units uniformly leads to better scores suggesting that the u2S component can take benefit from extra details of the input to produce a more realistic output. HuBERT and CPC seem to be giving the best results, for both humans and models better capturing phonetic information than other models at equivalent bitrates.

More detailed results are in Table 2 separating the scores for the LJ and LS resynthesis, and adding extra automatic metrics (CER) and human metrics (human CER). On PER, we found a domain effect: resynthesizing input from LJ Speech yields lower PER than from LibriSpeech on all unsupervised models. From the viewpoint of the encoder, LJ Speech is out-of-domain; therefore, one would expect that the units are making more errors than for the trained LibriSpeech. On the other hand, the u2S component has learned from LJ Speech encoded with these units, and might have learned to compensate for these lower quality units. When LibriSpeech is offered as input, the u2S component cannot adapt to this nominally better input and ends up yielding lower quality outputs. This observation is worth further explorations, as other metrics like CER (using an LM) and human evaluations only replicated this for the models with the lowest score (like LogMel and wav2vec). The automatic PER and CER scores and the human MOS and CER scores, all correlate well with one another across the 4 × 3 models and baselines. Within the LJ or LS domain , the Pearson r ranged from .95 to .99; across domains it was less good (from .79 to .96) illustrating again the existence of a domain effect. Not shown here, we reached similar conclusions with our fitted-ASR metrics, but with less good scores and correlations. Table 2 also shows the results of the two toplines (original text+TTS and ASR+TTS). Interestingly, our best models come within 3% absolute in PER or CER compared to these toplines, are quite close to them in terms of MOS and even beat them in terms of human CER.

Results on the generation task

The upper mid and right cells of Figure 3 shows generation results averaging across the unconditional and conditional conditions, on automatic and human evaluations respectively. The main result is that there is both an effect of number of units and of system. As for resynthesis, 50 units is always worst, but contrary to resynthesis, 200 units is not always better. Overall, the results on generation are congruent with the idea that speech generation both requires good scores on language modeling and on speech synthesis. The best results for a particular model are then a compromise between the number of units that give both scores to either of these tasks. In terms of systems, the best one here is HuBERT. Regarding human evaluations, they show similar patterns with a clear dispreference for 50 units, and either 100 or 200 being better.

Detailed results are shown in Table 3 with separate statistics for conditional and unconditional generation and additional results with PPX@o-VERT and VERT@o-PPX. As expected, the perplexity metric improved with prompts, but not the diversity score. The human results are congruent with the automatic scores, although they tend to prefer more units, perhaps showing that they cannot fully dissociate their judgment of meaning from their judgment of intelligibility. The three metrics correlate well with one another (r between .86 and .99) and correlate with their counterpart across task (prompted vs. unprompted: r between .82 and .99). Human evaluations correlated well with the automatic metrics (AUC: r=.87; PPX: r=.92; VERT: r=0.75).

Results for zero-shot probe metrics

In Table 4, we show the results for zero-shot metrics across the different models and baselines. Overall, the performances depend on the linguistic levels while remaining above chance. While performances are excellent at the acoustic level (6.5% error for the best model on ABX-across), they are intermediate at the lexical level (31.3% error for the best model on spot-the-word). Not shown, the syntactic test is close to chance (42% error for the best model on the sBLIMP test). These values are worse than the ASR-topline (3.1% and 29%, for lexicon and syntax resp.), showing room for improvement.

The metrics correlate well: the ABX score predicts the lexical score (r = 0.85) and the syn- tax score (r = 0.71). Across the different models, CPC gets the best units (ABX score) and Hu-BERT gets the best LM scores. In addition, we see a clear effect of number of units (Figure 3). For wav2vec, the performances on all metrics increase with more units, whereas, for CPC and HuBERT a U-shaped pattern emerges on most metrics, with best scores for units of intermediate sizes. It is interesting that the models with the highest bitrate do not always have the best results. This means that encoding too much acoustic information can be detrimental to linguistic encoding in the uLM. See Appendix Section 7.1 showing that ABX has good correlations with automatic and human metrics (r > .88).

Discussion and Conclusion

We introduced Generative Spoken Language Modeling as a new unsupervised task bridging the gap between speech and natural language processing and related it conceptually to previously studied unsupervised tasks: Acoustic Unit Discovery, Spoken Language Modeling, Discrete Speech Resynthesis and Text Generation. We introduced a suite of metrics, baselines, and first results on Lib-rilight that sets the playing field for future work.

For comparability, we open source our evaluation stack and the best of our baseline models.

Our main contributions are as follows:

(1) we established a set of easy to use automatic ASRbased metrics for model comparison at two critical levels for this task: intelligibility of the speech output and meaningfulness in terms of higher linguistic content. We assessed the first through ASR-based PER and CER metrics; and the second using text-generation-based metrics (AUC for PPX/VERT). (2) We found that these two sets of metrics correlated well with human judgement and (3) that they can be approximated with their inference-mode counterparts, which are faster to computed using zero-shot probe tasks. (4) Applying these metrics to pipeline models based on current speech representation learning models and out-of-the-box LM and TTS components, we found that our basic premise is fulfilled: it is possible to train a language model from quantized units derived from audio and using it to generate new speech. The generated speech is English sounding, with recognizable phonemes and words and locally acceptable syntax (see transcribed ex-Table 3: Results on the generation task for three unsupervised models plus the LogMel baseline and 3 unit sizes. PPX@-o-VERT and VERT@o-PPX are reported as PPX and VERT. '-' : missing or non calculable results. Human MMOS are also provided (the 95% confidence interval was on average .29 for uncond. and .61 for cond. amples in the Appendix and audio snippets here: https://speechbot.github.io/gslm). Our automatic metrics confirm the quality of the representations and outputs at the acoustic/phonetic level, but show that improvements are needed at the language level. It is to be expected that performance will increase with larger training sets beyond our 6k hours, as has been noted in the case of text. (5) We also uncovered specific issues regarding the number of quantized units. For speech resynthesis, the optimum number of units was always 200 by a large margin, reflecting the well known bitrate/intelligibility trade-off [START_REF] Dunbar | The Zero Resource Speech Challenge 2019: TTS without T[END_REF]. However, for language modeling, this was not necessarily the case, as the more detailed acoustic information may introduce too numerous phonetic details that have no impact at the level of lexical and syntactic representations. ( 6) Finally, we found that the choice of units also affected the temperature parameter which is used to control the trade-off between quality and diversity in text-based language model. To address this effect, we proposed a method to normalize the temperature by using an oracle text to build perplexity and diversity anchor points. Obviously, this is only a first step towards building textless NLP applications that could be applied to any language, even low resource ones. To reach this long term goal, three important challenges need to be addressed.

First, even though we did compare three different encoders and obtained different results we cannot conclude that one encoder is definitely su- and to take in another path and to take in another path and to take in another path and to take in another path and to take in another path and to take in another path and take in a Low temperature samples (obsessive zone) 0.7 chapter nineteen of the life of the upper part of the ocean this is ali bravos recording only bravos recordings are in the public domain i for more information or to volunteer 0.7 this is a lipper vox are courting oliver vox or courting are in the public domain for afraid art to volunteer pleases it lipper vox dot or this Mid temperature samples 1.0 but it is attendant from the people to defend himself from this information pride of the potential in criminal activity a curiosity and impetuosity of the world a war soon acquired 1.0 finally we ought to have a strong plan a without positively the best type of the public with which we ascend it or extend it our business and as we are a persons of the most strong designs and other affairs of the case we samples (babble zone) 1.5 ation of pure blue he said at once a licking streamy at her warm spot of half performed note was a raging oath let it as bir of amole in mood strolling er crass 1.5 at the swing here as to motions out of the events not time and abe he was any stump headed and flow any he's the kiln are tama why do ye take the floor perior to the others. Our point here was merely to use previously published pretrained encoders, and study systematically the effect of number of units on these encoders. A fuller study including a wider set of encoders and a proper hyperparameter search (including the selection of the embedding layer and the clustering algorithm) would be needed in order to determine which of them is most appropriate for speech generation. Second, it is to be expected that to further improve generation results, more needs to be done than applying this pipeline to larger training sets. Contrary to text, speech unfolds through time and varies continuously in phonetic space. Speech also contains multilayered representations (phonetic, prosodic, speaker identity, emotions, background noise, etc.). However, both our TTS and our LM were out-of-the-box systems typically used for text applications. More work is needed to adapt these architectures to the richness and variability of the speech signal (see Polyak et al., 2021a, for first steps towards integrating prosody into discrete units). The metrics and baselines we introduced here provide landmarks against which we will measure future progress.

Third, the automatic metrics that we defined here depend on textual resources to build the evaluation ASR and LM models, and on linguistic resources to build the zero-shot metrics. How could this ever be applied to low-resource languages? Note that the linguistic resources we require are used only for model selection, not model training. Our metrics allow for fast iterations in architecture and hyperparameter search, but the overall algorithm is totally unsupervised. Therefore, an important next step is to extend this work to other languages, in order to find a common architecture/hyperparameter set that gives good results in held out languages (high or low resource). The hope is that once good learning models are tuned using a diverse sample of high resource languages, the same models could be deployed in languages where no such resources are available, and work in a purely unsupervised fashion.

Appendix

Zero-shot metrics correlation results

In Figure A1, we present the Pearson correlations between the zero-shot metrics and the human and automatic metrics on downstream tasks. The fact that the ABX metric correlates well with these downstream metrics makes it a useful proxy metric for preliminary model and unit size selection, as it is much less costly than generating TTS output and running human or ASR evaluations.

Effect of temperature on outputs

In this section, we describe preliminary experiments we conducted to test the effects of temperature on the generated outputs. As shown in Table A1, the temperature defined qualitatively 4 operating zones. With the lowest temperature, we get repetitive outputs, where the system keeps repeating the same few words. At a slightly higher temperature, the system outputs complete sentences, but they are sampled from a narrow set of topics. At the highest temperature, the system utters an unstructured bag of words. In the mid-temperature range, we observe relatively coherent and varied outputs. This is the range to select for our systems. As described in Figure 2, the lowest bound was set by using the oracle PPX (temperature range between 0.2 and 0.65. across unsupervised models) and the highest bound by using the oracle VERT (temperature range between 1.1 to 1.4). In Figure S1 we present human opinion results for samples from these two temperatures, plus an extra mean temperature falling in between. Humans typically preferred the lower temperature.

In Figure A3, we illustrate the continuation method for selecting a single temperature for hu- man meaningfulness judgments in a model-neutral way as explained in Section 3.3. It consists in generating possible continuations of each prompt and computing the BLEU-2 score8 with oracle continuation. The temp@cont temperature is defined as the temperature maximizing this score. Computing these estimates with 10 continuations gave continuation temperatures varying between 0.5 and 0.9 across models and unit sizes. These are the temperatures we used for the MMOS results reported in the main paper.

Table S1: Full results using fitted ASR and Phoneme Recognition models of three unsupervised unit discovery models as a function of number of quantized units on 4 sets of metrics: PER and CER without LM & Lexicon (first 2 columns), and WER and CER with LM & Lexicon (last 2 columns). For comparison we also show the metrics for our supervised topline system. Table S2: Full Human evaluation MMOS results for three unsupervised models and 3 unit sizes. For each model and unit size, samples were generated using three different temperatures corresponding to: VERT, PPX and the average temperature of both.

Systems

Systems

Figure 1 :

 1 Figure 1: Setup of the baseline model architecture, tasks and metrics.

Figure 2 :

 2 Figure 2: Comparison of diversity and perplexity of the generated speech. We plot VERT vs. Median perplexity. The blue diamond corresponds to the oracle reference point. It defines two cut-offs on the curve: VERT @oracle-PPX and PPX @oracle-VERT. The green area corresponds to the AUC metric.

7

  For example, a pseudo-text 10 11 11 11 21 32 32 32 21 becomes 10 11 21 32 21.

Figure A1 :

 A1 Figure A1: Patterns of correlations between the zero-shot metrics and the automatic and human metrics. Color scale indicates strength of the Pearson correlation coeficient (we used negative MOS and MMOS to enforce less is better for all metrics).

Figure A2 :

 A2 Figure A2: MMOS for unconditional (no prompt) and conditional generated speech sampled at the three reference temperatures (oracle VERT, oracle PPX, and average temperature) (preliminary experiments).

Figure A3 :

 A3 Figure A3: Method for selecting the continuation temperature for MMOS judgements.

Figure S1 :

 S1 Figure S1: MMOS for unconditional (no prompt) and conditional generated speech sampled at the three reference temperatures (oracle VERT, oracle PPX, and average temperature).

Figure S2 :

 S2 Figure S2: Patterns of correlations between automatic and human metrics for generation. Color scale indicates strength and direction of the Spearman correlation coeficient.

Table 2 :

 2 Results on the resynthesis task for 3 unsupervised models plus one LogMel baseline and 3 unit sizes. Bitrates are in bit/sec, PER are for a pretrained phone recognition model without lexicon and LM, CER are derived from a full ASR model (lower is better). Human MOS (upper is better) and CER (computed from transcription, lower is better) are provided (the 95% confidence interval was on average .32 for MOS and 1.8 for human CER)

	Systems			End-to-end ASR-based metrics		Human Opinion
	S2u	Nb	Bit-	PER↓ PER↓ CER↓ CER↓ MOS↑ MOS↑ CER↓ CER↓
	architect.	units rate	(LJ)	(LS)	(LJ)	(LS)	(LJ)	(LS)	(LJ)	(LS)
	Toplines
	original wav			-	-	-	-	4.83	4.30	8.88	6.73
	orig text+TTS			7.78	7.92	8.87	5.14	4.02	4.03	13.25 10.73
	ASR + TTS	27		9.45	8.18	9.48	5.30	4.04	4.06	15.98 11.56
	Baselines
	LogMel	50	214.8 27.72 49.38 27.73 52.05	2.41	2.07	43.78 66.75
	LogMel	100 292.7 25.83 45.58 24.88 48.71	2.65	2.01	37.39 62.72
	LogMel	200 373.8 19.78 45.16 17.86 46.12	2.96	2.16	23.33	62.6
	Unsupervised
	CPC	50	159.4 10.87 17.16 10.68 12.06	3.63	3.51	13.97 19.92
	CPC	100 213.1 10.75 15.82	9.84	9.46	3.42	3.68	13.53 14.73
	CPC	200 279.4	8.74 14.23	9.20	8.29	3.85	3.54	9.36	14.33
	HuBERT-L6	50	125.7 11.45 16.68 11.02 11.85	3.69	3.49	14.54 13.14
	HuBERT-L6	100 168.1	9.53 13.24	9.31	7.19	3.84	3.68	13.02 11.43
	HuBERT-L6	200 211.3	8.87 11.06	8.88	5.35	4.00	3.85	11.67 10.84
	wav2vec-L14	50	141.3 24.95 33.69 25.42 32.91	2.45	2.87	46.82	54.9
	wav2vec-L14 100 182.1 14.58 22.07 13.72 17.22	3.50	3.32	23.76	28.1
	wav2vec-L14 200 226.8 10.65 16.34 10.21 10.50	3.83	3.51	13.14 15.27

Table 4 :

 4 Results for zero-shot probe metrics for 3 unsupervised models plus one LogMel baseline and 3 unit sizes. ABX within and across speakers, spot-the-

	).

Table A1 :

 A1 Unconditional uLM (trained on CPC-100 units) samples, transcribed by ASR, at different temperatures.

	Temp Example
	Very low temperature samples (stuttering zone)
	0.3	the property by james resell red for liberata or by jason downy the property by jason
		downy the property the property the property the property
	0.3

Higher self-BLEU scores indicate lower diversity of the produced text.

We use a LARGE wav2vec 2.0 model, trained on LibriSpeech-960h with CTC loss from scratch. Its decoder uses the standard KenLM 4-gram language model.

github.com/pytorch/fairseq/.../language_model

www.zerospeech.com

We used NLTK to compute BLEU[START_REF] Bird | Natural language processing with python[END_REF].

Acknowledgments

We thank Michael Auli and Alexis Conneau for their useful input on wav2vec, and Lior Wolf, Pierre Emmanuel Mazaré and Gargi Gosh for their support for this project. We would also like to thank the reviewers and editors for their thorough review, and constructive feedback.

Supplementary Materials

Here, we provide supplementary information not appearing in the TACL version for lack of space.

S1 Implementation Details

This section provides information about model training.

S1.1 Speech Decoder

We train all speech decoder models referred in Section 4.3 on 8 32-GB GPUs using data distributed training with a batch size of 32 per GPU for 500 epochs. We compute validation loss every 5000 steps, and select model with the lowest loss. For chunking (described in Section 4.3), we initialize the chunk size to 50 and increment it by 5 per epoch. Note that we only do chunking of Unit-To-Speech scenario only.

S1.2 Training of ASR Models for evaluation

S1.2.1 Frozen ASR Model

The frozen ASR model is trained using LARGE wav2vec model architecture with Connectionist Temporal Classification (CTC) loss [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF] from scratch (not using the pretrained model) on LibriSpeech 960hours dataset.

S1.2.2 Frozen Phoneme Recognition Model

The frozen phone recognition model is trained using BASE wav2vec model architecture with Connectionist Temporal Classification (CTC) loss [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF] from scratch on LibriSpeech 960hours dataset. We use g2p-en [START_REF] Park | [END_REF] for obtaining gold phoneme transcriptions.

S1.2.3 Fitted ASR Model

For speed of training, we design a SMALL version of wav2vec architecture with a Transformer encoder of 6 layers, 4 attention heads, embedding of size 256 and FFN of size 1024. The model is always trained with CTC loss using synthesized speech from the speech decoder.

S2 Supplementary Results

In Section S2.1, we present the full set of metrics used the resynthetized task, and in section S2.2, those for the generation task.

S2.1 Speech Resynthesis Results on fitted ASR metrics

In the table S1 we present the full set of metrics that we have used to evaluate synthesized speech using fitted ASR models. All the speech synthesis models are trained on LJSpeech and the synthesized speech is evaluated on LibriSpeech and LJ Speech. Note that we always use dev_clean set for LibriSpeech, and a random hold-out set of 1000 samples for LJ Speech that were not seen during training or validation.

S2.2 Generation Task

Table S2 shows the results of the meaninfulness opinion score (MMOS) detailing the three temperature settings (oracle PPX, oracle VERT and the average temperature between these two endpoints). As seen on Figure S1, human judgements are usually higher for the oracle PPX than the other two temperatures, and the pattern of results across systems is globally comparable across temperatures. Figure S2 shows the table of correlation coefficients across human and automatic metrics.


Adam Polyak
email: adampolyak@fb.com

Yossi Adi

Jade Copet

Eugene Kharitonov

Kushal Lakhotia

Wei-Ning Hsu

Abdelrahman Mohamed

Emmanuel Dupoux

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

Keywords: speech generation, speech resynthesis, selfsupervised learning, speech codec

We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found under the following link: speechbot.github.io/resynthesis.

Introduction

Learning unsupervised speech representations, both continuous and discrete, has seen a significant leap in performance following the recent success of Self-Supervised Learning (SSL) methods [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF][START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF][START_REF] Hsu | Hubert: How much can a bad teacher benefit ASR pre-training?[END_REF]. In the self-supervised setting, unlabeled inputs define an auxiliary task that can generate pseudo-labeled training data. This data can then be used to train a model using supervised techniques. The learned representations are often used for downstream tasks with a minimal amount of supervised data. For example, learning speech recognition with only 10 minutes of transcribed speech and 53K hours of untranscribed speech as in wav2vec 2.0 [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] and HuBERT [START_REF] Hsu | Hubert: How much can a bad teacher benefit ASR pre-training?[END_REF]. The learned self-supervised discrete representations also showed impressive performance on the conditional and unconditional spoken generative language modeling (GSLM) task [START_REF] Lakhotia | Generative spoken language modeling from raw audio[END_REF].

Despite its success, most studies on SSL for speech are focused on generating and evaluating the quality of the learned representations in the context of Automatic Speech Recognition (ASR). It remains unclear how suitable these representations are for speech synthesis. Moreover, in the context of expressive and controllable generation, it is unknown to what extent the speaker identity and F0 information are encoded in the learned representations.

Figure 1: The overall proposed speech resynthesis architecture. Three parallel encoders extract discrete representations from the raw input signal. These are then being used as a conditioning to reconstruct the signal using a decoder network.

Traditionally, speech synthesis and text-to-speech models produce Mel-spectrogram autoregressively given textual features as input. Next, a vocoder is applied to reconstruct the phase from the Mel-spectrogram (e.g., Griffin-Lim [START_REF] Griffin | Signal estimation from modified shorttime fourier transform[END_REF], WaveNet [START_REF] Oord | Wavenet: A generative model for raw audio[END_REF], WaveGlow [START_REF] Prenger | Waveglow: A flow-based generative network for speech synthesis[END_REF], or HiFi-GAN [START_REF] Kong | Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis[END_REF]). In this study, we suggest using the learned speech units as an input to a vocoder module with no spectrogram estimation. We additionally augment the learned units with quantized F0 representation and a global speaker embedding. Figure 1 presents the overall proposed method. This allows the evaluation of the learned units with respect to speech content, speaker identity, and F0 information, as well as better control the audio synthesis. We experiment with signal reconstruction, voice conversion, and F0 manipulation using several datasets and encoder models. Finally, equipped with our previous findings, we demonstrate how the learned units can function as an ultra-lightweight speech codec. Following the proposed method, we can reach an encoding rate of 365 bits per second (bps) while being superior to the baseline methods by a significant margin, including lightweight and heavyweight codecs. Our Contribution:

(i) We demonstrate the usage of discrete speech units, learned in a self-supervised manner, for high-quality synthesis purposes (no Mel-spectrogram estimation); (ii) We provide an extensive evaluation of the SSL speech units from a synthesis point of view, i.e., signal reconstruction, voice conversion, and F0 manipulation; (iii) We build an ultralightweight speech codec from the obtained speech units.

Related Work

Unsupervised Speech Representation Learning Studies on unsupervised speech representation learning can roughly be di-arXiv:2104.00355v3 [cs.SD] 27 Jul 2021 vided into reconstruction and self-supervised learning methods. Auto-encoding is the common approach for signal reconstruction, where speech is first encoded into a low-dimensional latent representation, and then decoded back to speech. Various constraints can be imposed on the encoded space, such as temporal smoothness [START_REF] Ebbers | Hidden markov model variational autoencoder for acoustic unit discovery[END_REF], discreteness [START_REF] Van Den Oord | Neural discrete representation learning[END_REF], and hierarchy [START_REF] Hsu | Unsupervised learning of disentangled and interpretable representations from sequential data[END_REF].

SSL methods have shown remarkable results for ASR [START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF][START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF], phoneme segmentation [START_REF] Kreuk | Self-supervised contrastive learning for unsupervised phoneme segmentation[END_REF], and GSLM [START_REF] Lakhotia | Generative spoken language modeling from raw audio[END_REF]. The authors in [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF][START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF] suggested training a convolutional neural network to distinguish true future samples from random distractor samples using a Contrastive Predictive Coding (CPC) loss function. Similar to CPC, the authors in [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] use an encoder and a predictor, which is trained contrastively to distinguish positive and negative samples. However, unlike [START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF] it discretizes and masks segments of the encoder's output. In HuBERT [START_REF] Hsu | Hubert: How much can a bad teacher benefit ASR pre-training?[END_REF], the model is trained with a masked prediction task similar to BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] but with masked continuous audio signals.

Speech Resynthesis

Recent advancements in neural-based vocoders enabled generating natural-sounding speech and music [START_REF] Oord | Wavenet: A generative model for raw audio[END_REF][START_REF] Prenger | Waveglow: A flow-based generative network for speech synthesis[END_REF][START_REF] Kong | Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis[END_REF]. These are often conditioned on the log Melspectrogram for the generation process. Recently, unsupervised learning of low bitrate speech representations was explored under the Zero-Resource Challenge [START_REF] Dunbar | The Zero Resource Speech Challenge 2019: TTS Without T[END_REF][START_REF]The Zero Resource Speech Challenge 2020: Discovering Discrete Subword and Word Units[END_REF]. Vector-Quantized Variational Auto-Encoder (VQ-VAE) [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] employs a learned fixed-sized codebook, decoded by a WaveNet model for speech synthesis. In [START_REF] Eloff | Unsupervised Acoustic Unit Discovery for Speech Synthesis Using Discrete Latent-Variable Neural Networks[END_REF] the authors proposed a VQ-VAE model followed by an FFTNet vocoder model [START_REF] Jin | Fftnet: A realtime speaker-dependent neural vocoder[END_REF]. The authors in [START_REF] Tjandra | Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge[END_REF] suggested using transformer [START_REF] Vaswani | Attention is all you need[END_REF] together with a VQ-VAE model for unsupervised unit discovery, and in [START_REF] Van Niekerk | Vector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge[END_REF] they combine vector quantization with contrastive predictive coding for acoustic unit discovery. The study by [START_REF] Zhao | Improved prosody from learned f0 codebook representations for vq-vae speech waveform reconstruction[END_REF] is the closest to our work. In which, the authors suggest training a VQ-VAE with two encoders, one for the waveform and the other for F0. The authors demonstrated how such modeling improves overall generation quality. In contrast, we study SSL-based speech encoders and empirically show these representations are better disentangled, and apply them as an ultra-low bitrate speech codec. Another line of work suggests using intermediate representations obtained from an ASR acoustic model. These representations are being used together with the identity and prosodic information for voice conversion [START_REF] Polyak | TTS Skins: Speaker Conversion via ASR[END_REF][START_REF] Polyak | Unsupervised Cross-Domain Singing Voice Conversion[END_REF][START_REF]High fidelity speech regeneration with application to speech enhancement[END_REF]. Unlike all of the above, we suggest synthesizing speech directly from the discrete units. Moreover, the resynthesis process sheds light on the encoded information in each of the evaluated representations.

Speech Codec

Speech codecs typically employ a carefully hand-engineered pipeline combining an encoder and a decoder which is influenced by speech production physics to remove redundancies in the data and yield a compact bitstream. Low bitrate parametric speech codecs have long been studied [START_REF] Atal | Speech analysis and synthesis by linear prediction of the speech wave[END_REF], but their quality has been severely limited. Despite some advances [START_REF] Griffin | A new model-based speech analysis/synthesis system[END_REF][START_REF] Mccree | A 2.4 kbit/s melp coder candidate for the new us federal standard[END_REF], modeling the excitation signal has remained a challenge. Neural speech codecs have been recently proposed and demonstrated promising results [START_REF] Kleijn | Wavenet based low rate speech coding[END_REF][START_REF] Lim | Robust low rate speech coding based on cloned networks and wavenet[END_REF]. In [START_REF] Valin | A real-time wideband neural vocoder at 1.6 kb/s using lpcnet[END_REF] an LPC-Net [START_REF]Lpcnet: Improving neural speech synthesis through linear prediction[END_REF] vocoder was conditioned on hand-crafted features and a uniform quantizer. In [START_REF] Gârbacea | Low bit-rate speech coding with vq-vae and a wavenet decoder[END_REF] a WaveNet model was conditioned on discrete units obtained from a VQ-VAE model, while in [START_REF] Skoglund | Improving opus low bit rate quality with neural speech synthesis[END_REF] the Opus codec [START_REF] Valin | Definition of the opus audio codec[END_REF] was fed to WaveNet.

Method

The proposed architecture is comprised of three pre-trained and fixed encoders, namely: (i) content encoder; (ii) F0 encoder; (iii) speaker identity encoder; and a decoder network. The first two encoders extract discrete representation from the raw au-dio while the latter extracts a single global representation. The overall architecture is depicted in Figure 1.

Encoders

Denote the domain of audio samples by X ⊂ R. The representation for a raw signal is therefore a sequence of samples x = (x1, . . . , xT ), where xt ∈ X for all 1 ≤ t ≤ T .

Content Encoder

The input to a content encoder network, Ec, is a speech utterance, x, and the output is a sequence of spectral representations sampled at a low frequency as follows Ec(x) = (v1, . . . , v T ). We evaluated three state-of-the-art unsupervised representation learning functions as Ec. Specifically, we experimented with: (i) CPC [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF] which attempts to predict the future states of the encoder based on the past and optimizes a contrastive loss comparing the actual future from that of random sequences; (ii) HuBERT [START_REF] Hsu | Hubert: How much can a bad teacher benefit ASR pre-training?[END_REF] which was trained with a masked prediction task similar to BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] on masked continuous audio signals as inputs. The targets are obtained through clustering of raw speech features or learned features from earlier iterations; (iii) and VQ-VAE [START_REF] Van Den Oord | Neural discrete representation learning[END_REF] which performs similarly to a Variational Auto Encoder [START_REF] Kingma | Auto-encoding variational bayes[END_REF] where the encoder's output is discrete rather than continuous.

Since the representations learned by CPC and HuBERT are continuous, a k-means algorithm is applied over the models' outputs to generate discrete units, denoted as zc = (z1, . . . , zL). Each element zi in zc is a positive integer, zi ∈ {0, 1, .., K} for 1 ≤ i ≤ L, where K is the number of discrete units. We did not follow the same approach for VQ-VAE as its representations are already quantized.

F0 Encoder

To generate low frequency discrete F0 representation, zF 0 = (z1, . . . , z L ), a separate encoder, EF 0 , is applied over the F0 extracted from the input signal. Each element in zF 0 is an integer zs ∈ {0, 1, .., K }, where K is the encoder dictionary size. The YAAPT [START_REF] Kasi | Yet another algorithm for pitch tracking[END_REF] algorithm is used to extract the F0 from the input signal, x, generating p = (p1, . . . , p T ).

EF 0 is trained using the VQ-VAE framework. VQ-VAE employs a convolutional encoder, EF 0 , a bottleneck with a learned codebook C = (e1, . . . , e K ), where each item in C is a 128-dimensional vector, and a decoder DF 0 . The encoder extracts a sequence of latent vectors EF 0 (p) = (h1, . . . , h L ) from the raw audio, where hi ∈ R 128 , for all 1 ≤ i ≤ L . Then, the bottleneck maps each latent vector to its nearest vector in the codebook C. The embedded latent vectors are then being fed into the decoder DF 0 (ez 1 , . . . , ez L ) = p which reconstructs the original F0 signal. Similar to [START_REF] Dhariwal | Jukebox: A generative model for music[END_REF], we use Exponential Moving Average updates to learn the codebook and employ random restarts for unused embeddings.

To generate zF 0, we use the indices of the mapped latent vectors rather than the vectors.

Speaker Encoder

Lastly, a speaker encoder, E spk , is used to extract speaker embedding. A pre-trained speaker verification model similar to the one proposed in [START_REF] Heigold | End-to-end text-dependent speaker verification[END_REF] is used. Formally, E spk gets as input the speech utterance, x, extracts the Mel-spectrogram and outputs a d-vector speaker representation, denoted as z spk ∈ R 256 . We experimented with learning the speaker embedding via a lookup-table. Although such method performs slightly better, it is limited to speakers seen during training.

Decoder

A neural vocoder is employed to decode the speech signal from the discrete representation. This study considers the decoder to be a modified version of the HiFi-GAN [START_REF] Kong | Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis[END_REF] neural vocoder.

The HiFi-GAN architecture is comprised of a generator, G, As an input, the generator receives the encoded representation (zc, zF 0 , z spk ). The discrete content sequence, zc, and the discrete pitch sequence, zF 0 , are converted to a continuous representation via LU Tc and LU TF 0 accordingly. The sequences are up-sampled and concatenated together. The speaker embedding, z spk , is concatenated to each frame in the up-sampled sequence.

The discriminator comprises two networks, a Multi-Period Discriminator (MPD) and a Multi-Scale Discriminator (MSD). The MPD consists of multiple sub-discriminators operating on equally spaced samples from the input signal. The period sub-discriminators differ from each other based on the space between the samples. Similar to [START_REF] Kong | Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis[END_REF], the MPD employs a total of five-period discriminators with a period hops of [START_REF] Schneider | wav2vec: Unsupervised Pre-Training for Speech Recognition[END_REF][START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF][START_REF] Lakhotia | Generative spoken language modeling from raw audio[END_REF][START_REF] Oord | Wavenet: A generative model for raw audio[END_REF][START_REF] Van Den Oord | Neural discrete representation learning[END_REF]. Multi-scale discriminator (MSD) employs multiple sub-discriminators operating at different scales of the input signal. Specifically, we use three scales: the original input scale, ×2 downsampled scale, and ×4 downsampled scale. Overall each sub-discriminator Dj is tasked with minimizing the following,

L adv (Dj, G) = x ||1 -Dj( x)|| 2 2 , LD(Dj, G) = x [||1 -Dj(x)|| 2 2 + ||Dj( x)|| 2 2 ], (1)
where x = G(LU Tc(zc), LU TF 0 (zF 0 ), z spk ), is the resynthesized signal from the encoded representation. Additionally, two terms are added to the loss function. The first one is a reconstruction term computed between the Melspectrogram of the input signal and the generated signal,

Lrecon(G) = x ||φ(x) -φ( x)||1, (2)
where φ is a spectral operator computing Mel-spectrogram. The second term is a feature-matching loss [START_REF] Larsen | Autoencoding beyond pixels using a learned similarity metric[END_REF] which measures the distance between discriminator activations of the real signal and those of the resynthesized signal, where ψi is an operator which extracts the activations of the discriminator i-th layer, Mi is the number of features in layer i, and R is the total number of layers in Dj.

L f m (Dj, G) = x R i=1 1 Mi ||ψi(x) -ψi( x)||1, (3)
The final loss with respect to the sub-discriminators composing discriminator D and generator G is:

L multi G (D, G) = J j=1 [L adv (G, Dj) + λ f m L f m (G, Dj)], + λrLrecon(G), L multi D (D, G) = J j=1 LD(G, Dj), (4)
where we set λ f m = 2 and λr = 45.

Results

Our results cover three different settings: (i) speech reconstruction experiments; (ii) speaker conversion and F0 manipulation; (iii) bitrate analysis with subjective tests for speech codec evaluation. We employ two datasets: LJ [START_REF] Ito | The lj speech dataset[END_REF] single speaker dataset and VCTK [START_REF] Veaux | CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit[END_REF] multi-speaker dataset. All datasets were resampled to a 16kHz sample rate. Implementation Details We follow the same setup as in [START_REF] Lakhotia | Generative spoken language modeling from raw audio[END_REF]. For CPC, we used the model from [START_REF] Rivière | Towards unsupervised learning of speech features in the wild[END_REF], which was trained on a "clean" 6k hour sub-sample of the LibriLight dataset [START_REF] Kahn | Libri-light: A benchmark for asr with limited or no supervision[END_REF][START_REF] Rivière | Towards unsupervised learning of speech features in the wild[END_REF]. We extract a downsampled representation from an intermediate layer with a 256-dimensional embedding and a hop size of 160 audio samples. For HuBERT we used a BASE 12 transformerlayer model trained for two iterations [START_REF] Hsu | Hubert: How much can a bad teacher benefit ASR pre-training?[END_REF] on 960 hours of Lib-riSpeech corpus [START_REF] Panayotov | Librispeech: an asr corpus based on public domain audio books[END_REF]. This model downsamples the raw audio ×320 into a sequence of 768-dimensional vectors. Similarly to [START_REF] Lakhotia | Generative spoken language modeling from raw audio[END_REF], activations were extracted from the sixth layer.

For CPC and HuBERT, the k-means algorithm is trained on LibriSpeech clean-100h [START_REF] Panayotov | Librispeech: an asr corpus based on public domain audio books[END_REF] dataset to convert continuous frames to discrete codes. We quantize both learned representations with K = 100 centroids. Leading to a bitrate of 700bps for CPC and 350bps for HuBERT.

Similarly to CPC models, we trained the VQ-VAE content encoder model on the "clean" 6K hours subset from the Libri-Light dataset. We use an encoder operating on the raw signal to extract discrete units, similar to [START_REF] Dhariwal | Jukebox: A generative model for music[END_REF]. In addition, "random restarts" were performed when the mean usage of a codebook vector fell below a predetermined threshold. Finally, we used HiFiGAN (architecture and objective) as the decoder instead of a simple convolutional decoder, as it improved the overall audio quality. This model encodes the raw audio into a sequence of discrete tokens from 256 possible tokens [START_REF] Gârbacea | Low bit-rate speech coding with vq-vae and a wavenet decoder[END_REF] with a hop size of 160 raw audio samples. The VQ-VAE discrete code operates at a bitrate of 800bps. We additionally experimented with 100 discrete units for VQ-VAE, however results were the best for 256. This finding is consistent with [START_REF] Gârbacea | Low bit-rate speech coding with vq-vae and a wavenet decoder[END_REF].

The speaker verification network uses the architecture proposed in [START_REF] Heigold | End-to-end text-dependent speaker verification[END_REF]. It was trained on the VoxCeleb2 [START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF] dataset, achieving a 7.4% Equal Error Rate (EER) for speaker verification on the test split of the VoxCeleb1 [START_REF] Nagrani | Voxceleb: a largescale speaker identification dataset[END_REF] dataset.

Only a single F0 representation is considered across all evaluated models, trained on the VCTK dataset. The F0 is extracted from the raw audio using a window size of 20ms and a 5ms hop. As a result, the F0 sequence is sampled at 200Hz. The quantization described at Sec. 3, is applied using an F0 codebook of K = 20 tokens and an encoder that downsamples the signal by ×16. Hence, the discrete F0 representation is sampled at 12.5Hz, leading to a bitrate of 65bps. The final bitrate of the evaluated codecs is the sum of the pitch code bitrate with the content code bitrate. Evaluation Metrics We consider both subjective and objective evaluation metrics. For subjective tests, we report the Mean Opinion Scores (MOS). In which human evaluators rate the naturalness of audio samples on a scale of 1-5. Each experiment, included 50 randomly selected samples rated by 30 raters. For objective evaluation, we consider: (i) Equal Error Rate (EER) as an automatic speaker verification metric obtained using a pre-trained speaker verification network. We report EER between test utterances and enrolled speakers; (ii) Voicing Decision Error (VDE) [START_REF] Nakatani | A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environments[END_REF], which measures the portion of frames with voicing decision error; (iii) F0 Frame Error (FFE) [START_REF] Chu | Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend[END_REF], measures the percentage of frames that contain a deviation of more than 20% in pitch value or have a voicing decision error; (iv) Word Error Rate (WER) and Phoneme Error Rate (PER), proxy metrics to the intelligibility of the generated audio. We used a pre-trained ASR network [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] on both reconstructed and converted samples to calculate both metrics.

Reconstruction & Conversion We start by reporting the reconstruction performance. Results are summarized in Table 1. When considering the intelligibility of the reconstructed signal HuBERT reaches the lowest PER and WER scores across all models, where both CPC and HuBERT are superior to VQ-VAE. However, when considering F0 reconstruction VQ-VAE outperforms both HuBERT and CPC by a significant margin. This results are somewhat intuitive, bearing in mind VQ-VAE objective is to fully reconstruct the input signal. In terms of subjective evaluation, all models reach similar MOS scores, with one exception of CPC on LJ.

To better evaluate the disentanglement properties of each method with respect to speaker identity and F0, we conducted an additional set of experiments aiming at speaker conversion and F0 manipulation. For voice conversion, we converted each test utterance into five random target speakers. Next, we employed a speaker verification network, which extracts d-vector representation to evaluate speaker-converted utterances' similarity to real speaker utterances (low error-rate indicates good conversion), providing measurement to the speaker identity's disentanglement from the evaluated coding method. The errorrate is reported between converted test utterances and enrolled speakers. For the LJ speech single speaker dataset, we converted samples from the VCTK dataset to the single speaker and enrolled all VCTK speakers together with the single speaker. Results are summarized in Table 2 (left). Unlike resynthesis results, on voice conversion CPC and HuBERT outperform VQ-VAE on both LJ and VCTK datasets, indicating VQ-VAE contains more information about the speaker in the encoded units, hence producing more artifacts. Notice, this also affects WER,

VQ-VAE 800bps

Figure 2: MUSHRA subjective quality results as a function of bitrate per second. Purple dots denote the baseline methods, and green dots the proposed SSL based method.

PER, and the overall subjective quality (MOS).

Next, to evaluate the presence of F0 in the discrete units, we flattened the F0 units before synthesizing the signal and calculated VDE and FFE with respect to the original F0 values. F0 flattening was done by setting the speakers' mean F0 value across all voiced frames. In this experiment, we expected units that contain F0 information to be better at F0 reconstruction over disentangled units. Results are summarized in Table 2 (right). Notice VQ-VAE can still reconstruct the F0 almost at the same level as when using the original F0 as conditioning (5.2 vs 7.03, and 5.59 vs 7.8), in contrast to CPC and HuBERT. Speech Codec Our final experiment evaluates the obtained speech units as a low bitrate speech codec. We use a subjective MUSHRA-type listening test [START_REF] Series | Method for the subjective assessment of intermediate quality level of audio systems[END_REF] to measure the perceived quality of the proposed speech codec with regard to its bitrate constraints. In MUSHRA evaluations, listeners are presented with a labeled uncompressed signal for reference, a set of test samples to rate, a copy of the uncompressed reference, and a low-quality anchor. Listeners are asked to rate each test utterance and the copy of the uncompressed reference with respect to the labeled reference in a scale of 1-100.

The experiment is performed on the VCTK dataset [START_REF] Veaux | CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit[END_REF]. For evaluation, we used 20 utterances from 5 speakers. The set of speakers in the test data is disjoint with those in the training data. For this experiment, HuBERT models with 50, 100, and 200 units were trained as described in Sec. 4. For comparison, we included other speech codecs in our evaluation: Opus [START_REF] Valin | Definition of the opus audio codec[END_REF] wideband at 9 kbps VBR, Codec2 [START_REF] Rowe | Codec 2-open source speech coding at 2400 bits/s and below[END_REF] at 2.4 kbps and LPC-Net [START_REF] Valin | A real-time wideband neural vocoder at 1.6 kb/s using lpcnet[END_REF] operating at 1.6 kbps. The LPCNet model was trained from scratch on the VCTK dataset following the experimental setup in [START_REF] Valin | A real-time wideband neural vocoder at 1.6 kb/s using lpcnet[END_REF]. The VQ-VAE model employs the HiFiGAN decoder trained on the LibriLight dataset to match the amount of data reported in [START_REF] Gârbacea | Low bit-rate speech coding with vq-vae and a wavenet decoder[END_REF]. We compressed the anchor sample with Speex [START_REF] Valin | Speex: A free codec for free speech[END_REF] at 4 kbps as a low anchor. Fig. 2 depicts the results. HuBERT with 50 units reaches the best MUSHRA score while its bitrate is only 365bps, which is significantly lower than the baseline methods.

Conclusion

We applied self-supervised discrete representations for the task of speech resynthesis. Furthermore, we demonstrated the efficiency of disentangled representations for signal reconstruction, voice conversion, and F0 manipulation. Our evaluations shed light on the properties encoded by each method in the context of speech synthesis. Finally, we adapt the HuBERT speech representation as an ultra lightweight speech codec, providing superior subjective results than the baselines with lower bitrate.

Table 1 :

 1 Speech resynthesis results. MOS results are reported with 95% confidence interval. Except MOS for all other evaluation metrics the lower is the better. We did not report EER for LJ since it is a single speaker dataset. The generator is built from a set of look-up tables (LUT) that embed the discrete representation and a series of blocks composed of transposed convolution and a residual block with dilated layers. The transposed convolutions upsample the encoded representation to match the input sample rate, while the dilated layers increase the receptive field.

	Dataset Method	Content	F0		Speaker Overall Quality
			PER ↓ WER ↓ VDE ↓ FFE ↓ EER ↓	MOS ↑
		GT	6.93 5.60	-	-	-	4.33±0.20
	LJ	CPC	9.66 8.51	13.48 15.19	-	3.31±0.33
		HuBERT 9.52 6.96	13.09 15.00	-	3.66±0.33
		VQ-VAE 12.77 8.85	7.19 8.54	-	3.66±0.31
		GT	17.16 4.32	-	-	3.25	4.08±0.66
	VCTK	CPC	23.01 14.49	10.56 11.13	4.25	3.33±0.61
		HuBERT 19.66 11.44	9.77 10.43	5.79	3.41±0.66
		VQ-VAE 31.97 19.80	5.20 5.59	4.28	3.39±0.58
	and a set of discriminators, D.

Table 2 :

 2 Voice conversion & F0 manipulation results. MOS results are reported with 95% confidence interval. VDE, and FFE are reported for F0 manipulation while PER, WER, EER, and MOS are reported for voice conversion. Notice, for VDE, and FFE higher is the better since F0 was flattened.

	Dataset Method	Voice Conversion	F0 Manipulation
			PER ↓ WER ↓ EER ↓	MOS ↑	VDE ↑ FFE ↑
	VCTK	GT	17.16 4.32	3.25	4.11±0.29	-	-
		CPC	22.22 16.11	0.46	3.57±0.15	46.68 48.71
	LJ	HuBERT 19.09 12.23	0.31	3.71±0.24	39.20 48.42
		VQ-VAE 40.88 36.96	9.65	2.90±0.17	10.54 12.08
		CPC	23.58 15.98	4.83 3.42 ± 0.24	25.29 26.97
	VCTK	HuBERT 20.85 12.72	6.01 3.58 ± 0.28	23.46 26.67
		VQ-VAE 36.88 29.44	11.56 3.08 ± 0.34	7.03	7.80