Natalia Tomashenko

Brij Mohan

Lal Srivastava

Xin Wang

Emmanuel Vincent

Andreas Nautsch

Junichi Yamagishi

Nicholas Evans

Jose Patino

Jean-François Bonastre

Paul-Gauthier Noé

Massimiliano Todisco

The VoicePrivacy 2020 Challenge Evaluation Plan

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Given that speech data can qualify as personal data, it is thus of no surprise that efforts to develop privacy preservation solutions for speech technology are starting to emerge. The study of privacy is also gaining traction; the Security and Privacy in Speech Communication (SPSC) 1 Special Interest Group (SIG) of the International Speech Communication Association (ISCA) has recently been formed to champion research in the domain.

There are arguably two general solutions to preserve privacy in speech data: cryptography, and anonymization, also referred to as de-identification. Cryptographic solutions can render speech data unreadable and can only be read in plain text with access to a private key. Some techniques, e.g. homomorphic encryption, even support computation upon data in the encrypted domain. These methods are typically specific to the given application, are challenging to integrate within existing systems and almost always result in significant increases to computational complexity and/or communication overheads. As such, cryptographic solutions can be relatively inflexible and cumbersome. Even then, they can only be implemented with specialist cryptography expertise.

Anonymization is different. Instead of conserving and protecting sensitive data through encryption, anonymization techniques, e.g. [START_REF] Fang | Speaker anonymization using X-vector and neural waveform models[END_REF], have the goal of suppressing personally identifiable information within a speech signal, leaving all other aspects of the speech signal intact. Anonymization techniques can be highly flexible and can be integrated easily within existing systems. They usually have low computational complexity and rarely require any additional expertise outside of the speech domain. Despite the appeal of anonymization techniques and the urgency to address privacy concerns, there are currently only few solutions. In addition, a formal definition of anonymization is missing, the level of anonymization offered by some solutions is somewhat unclear, and there are no common datasets, protocols and metrics to support further work.

The VoicePrivacy initiative is spearheading the effort to develop privacy preservation solutions for speech technology. It aims to gather a new community to define the task and metrics and to benchmark initial solutions using the first common datasets and protocols. In following best practice, VoicePrivacy takes the form of a competitive challenge. Challenge participants are required to process a dataset of speech signals in order to anonymize them, while protecting the linguistic content and speech naturalness. The challenge will run from early 2020 and conclude with a special session/event held in conjunction with Interspeech 2020, at which challenge results will be made publicly available. This document describes plans for the challenge, the dataset, protocol and the set of metrics that will be used for assessment, in addition to evaluation rules and guidelines for registration and submission.

Challenge objectives

The grand objective of the VoicePrivacy challenge is to foster progress in the development of anonymization techniques for speech data. The specific technical goals are summarised as follows. They are to:

• facilitate the development of novel techniques which suppress speaker-discriminative information within speech signals;

• promote techniques which provide effective anonymization while protecting intelligibility and naturalness;

• provide a level playing field to facilitate the comparison of different anonymization solutions using a common dataset and protocol;

• investigate metrics for the evaluation and meaningful comparison of different anonymization solutions.

VoicePrivacy participants will be provided with a common datasets, protocols and a suite of software packages that will enable them to gauge anonymization performance. While it is likely that this will evolve in future challenges, no formal ranking of systems will be made for this first edition. This is because recommendations for the assessment of anonymization solutions form one of the technical goals.

Task

A privacy preservation task is typically formulated as a game involving one or more users who publish some data and an attacker (also called an adversary) who gains legal or illegal access to this data or to derived data and attempts to infer personal information about the users. To protect their privacy, the users publish data that contain as little personal information as possible while allowing one or more desired goals to be fulfilled. To infer personal information about the users, the attacker may use additional knowledge.

Considering speech data as an example, the definition of the privacy preservation task depends on the following scenario specifications [START_REF] Srivastava | Evaluating voice conversion-based privacy protection against informed attackers[END_REF]:

• the nature of the data (e.g., speech waveform, speech features, text, etc.)

• the pieces of information considered as personal (e.g., speaker identity, speaker traits, spoken contents, etc.)

• the desired goal(s) (e.g., human dialogue, multi-party human conversation, automatic speech recognition (ASR) training, emotion recognition, etc.)

• the data accessed by the attacker (e.g., a single utterance, multiple utterances, features computed from these utterances, an ASR model trained on these utterances, etc.)

• the additional knowledge available to the attacker (e.g., utterances previously released by the user, the measures adopted by users to protect their privacy, etc.)

Different scenario specifications lead to different privacy preservation methods from the user's point of view and different attack methods from the attacker's point of view.

Scenario

In VoicePrivacy 2020, we consider the following specific scenario, where the terms "user" and "speaker" are used interchangeably:

• Speakers want to hide their identity while allowing any desired goal to be potentially achieved.

• The attacker has access to a single utterance and wants to identify the corresponding speaker.

Anonymization task

In order to hide his/her identity, each speaker passes his/her utterances through an anonymization system before publication. The resulting anonymized utterances are called trial utterances. They sound as if they were uttered by another speaker, which we call pseudo-speaker which may be an artificial voice not corresponding to any real speaker.

The task of challenge participants is to develop this anonymization system. In order to allow any desired goal to be potentially achieved, this system should satisfy the following requirements:

• output a speech waveform,

• hide speaker identity as much as possible,

• distort other speech characteristics as little as possible,

• ensure that all trial utterances from a given speaker appear to be uttered by the same pseudospeaker, while trial utterances from different speakers appear to be uttered by different pseudospeakers.

The third requirement will be assessed via a range of utility metrics (also called usability metrics). Specifically, ASR performance using a model trained on clean (non-anonymized) data and subjective speech intelligibility and naturalness will be measured during the challenge (see Section 5.1 and 5.2), and additional desired goals including ASR training will be assessed in a post-evaluation stage (see Section 8).

The fourth requirement is motivated by the fact that, in a multi-party human conversation, each speaker cannot change his/her anonymized voice over time and the anonymized voices of all speakers must be distinguishable from each other. This will be assessed via a new, specifically designed metric in the post-evaluation stage.

Attack model

We assume that the attacker has access to various amounts of data:

• one or more anonymized trial utterances,

• possibly, several additional utterances for each speaker, which may or may not have been anonymized and are called enrollment utterances.

The attacker does not have access to the anonymization system applied by the user though. The level of protection of personal information will be assessed via a range of privacy metrics, including objective speaker verifiability metrics and subjective speaker verifiability and linkability metrics. These metrics assume different attack models:

• The objective speaker verifiability metrics assume that the attacker has access to a single anonymized trial utterance and several enrollment utterances. During the challenge, two sets of metrics will be computed, corresponding to the two situations when the enrollment utterances are clean or they have been anonymized (see Section 5.1). In the latter case, we assume that the utterances have been anonymized in the same way as the trial data using the same anonymization system, i.e., all enrollment utterances from a given speaker are converted into the same pseudo-speaker, and enrollment utterances from different speakers are converted into different pseudo-speakers. We also assume that the pseudo-speaker corresponding to a given speaker in the enrollment set is different from the pseudo-speaker corresponding to that same speaker in the trial set. In the post-evaluation stage, we will consider alternative anonymization procedures for the enrollment data, corresponding to stronger attack models (see Section 8).

• The subjective speaker verifiability metric assumes that the attacker has access to a single anonymized trial utterance and a single clean enrollment utterance (see Section 5.2).

• The subjective speaker linkability metric assumes that the attacker has access to several anonymized trial utterances (see Section 5.2).

Data

Several publicly available corpora are used for training, development and evaluation of speaker anonymization systems. They are comprised of subsets from the following corpora:

• LibriSpeech 2 [5]

Training data

For training an anonymization system the following corpora can be used: VoxCeleb-1,2, LibriSpeechtrain-clean-100, LibriSpeech-train-other-500, LibriTTS-train-clean-100 and LibriTTS-train-other-500. Summary statistics for these data are given in Table 1.

Development set

Subsets from two different corpora are provided as development sets for anonymization systems. The first one is a subset of the LibriSpeech-dev-clean dataset. The second one (denoted as VCTKdev ) is obtained from the VCTK corpus. As explained in Section 3, we split these datasets into a trial subset, which is to be anonymized, and an enrollment subset, which is to be anonymized with different pseudo-speakers in order to compute objective speaker verification metrics. The anonymized enrollment data shall be stored in a different folder than the clean enrollment data, since both clean and anonymized enrollment data will be used to computed two different sets of metrics. The statistics of these subsets are summarized in Table 2. For the LibriSpeech-dev-clean dataset, the speakers in the enrollment set are a subset of those in the trial set. For the VCTK-dev dataset, we created two subsets of trial utterances, denoted as common part and different part. Both include trials from the same set of speakers, but from disjoint subsets of utterances. The common part of the trials is composed of utterances #1 -24 in the VCTK corpus that are identical for all speakers: the elicitation paragraph6 (utterances #1 -5) and rainbow passage7 (utterances #6 -24). This part is meant for subjective evaluation of speaker verifiability/linkability in a text-dependent manner. The enrollment subset and the different part of the trials are composed of distinct utterances for all speakers (utterances with indexes ≥ 25).

Evaluation data

Similarly to the development data, subsets from two different corpora are used for evaluation. The first one is a subset of the LibriSpeech-test-clean dataset. The second one (denoted as VCTK-test) is obtained from the VCTK corpus. We split those datasets into enrollment and trial subsets as summarized in Table 3. For the VCTK dataset, we created two subsets of trial utterances, denoted as common part and different part, in a similar manner as for the development set.

Utility and privacy metrics

Following the attack model assumed in Section 3.3, we consider objective and subjective privacy metrics to assess speaker re-identification and linkability. We also propose objective and subjective utility metrics in order to assess the fulfillment of the user goals specified in Section 3.2. Specifically, we consider ASR performance using a model trained on clean data and subjective speech intelligibility and naturalness. Objective metrics will be computed by the participants themselves using the provided evaluation scripts and evaluation data, while subjective metrics will be computed the organizers using data provided by the participants.

Objective metrics

For objective evaluation of anonymization performance, two systems will be trained to assess the following characteristics: (1) speaker verifiability and (2) ability of the anonymization system to preserve linguistic information in the anonymized speech. The first system denoted as ASV eval is an automatic speaker verification (ASV) system, which produces log-likelihood ratio (LLR) scores. The second system denoted as ASR eval is an automatic speech recognition (ASR) system which outputs a word error rate (WER) metric.

Both ASR eval and ASV eval are trained on the LibriSpeech-train-clean-360 dataset using the Kaldi speech recognition toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF]. The statistics of this dataset are given in Table 4. Training of ASR eval and ASV eval and evaluation will be done with the provided recipes 8 . As shown in Figure 1, the anonymized development and evaluation data will be decoded using the provided pretrained system ASR eval and the word error rate (WER) will be calculated:

WER = N sub + N del + N ins N ref
with N sub , N del , and N ins , the number of substitution, deletion, and insertion errors, respectively, and N ref the number of words in the reference. This will be compared to the WER achieved on clean data. The WER is calculated only on the trial part of the development and evaluation datasets.

Speaker verifiability metrics

The ASV eval system for speaker verification is based on state-of-the-art x-vector speaker embeddings with a probabilistic linear discriminant analysis (PLDA) backend [START_REF] Snyder | X-vectors: Robust DNN embeddings for speaker recognition[END_REF] as implemented in Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF].

It will be applied as shown in Figure 2: The number of target and impostor trials in the development and evaluation datasets are given in Tables 5 and6, respectively. Log-likelihood-ratio cost function (C llr and C min llr ) In addition to the EER, an alternative performance measure, the log-likelihood-ratio cost function (C llr ), will be estimated. It was proposed in [START_REF] Brümmer | Application-independent evaluation of speaker detection[END_REF] as an application-independent 9 evaluation objective and is defined as follows:

C llr = 1 2   1 N tar i∈targets log 2 1 + e -LLRi + 1 N imp j∈impostors log 2 1 + e LLRj   ,
where N tar and N imp are respectively the number of target and impostor LLR values in the evaluation set.

C llr can be decomposed into a discrimination loss (C min llr ) 10 and a calibration loss (C llr -C min llr ) 11 [START_REF] Brümmer | Application-independent evaluation of speaker detection[END_REF]. The C min llr is estimated by optimal calibration using monotonic transformation of scores to their empirical LLR values. To obtain this monotonic transformation, the pool adjacent violators (PAV) to LLR algorithm is used [START_REF] Brümmer | Application-independent evaluation of speaker detection[END_REF][START_REF] Ramos | Cross-entropy analysis of the information in forensic speaker recognition[END_REF].

Subjective metrics

Several subjective metrics will be used in evaluation: speaker verifiability, speaker linkability, speech intelligibility, and speech naturalness. They will be evaluated using listening tests that will be carried out by the organizers as described below.

Subjective speaker verifiability

To evaluate subjective speaker verifiability, the following two approaches can be used. The first approach is to measure the speaker similarity between one anonymized trial utterance and one clean enrollment utterance through a large-scale crowdsourced evaluation, following the approach used in a recent study on speech anti-spoofing [START_REF] Wang | The ASVspoof 2019 database[END_REF]. Given one anonymized trial utterance and one randomly selected clean enrollment utterance from the same original speaker, the subjects will be instructed 9 In this context, application is defined as a set of prior probabilities and decision costs involved in the inferential process [START_REF] Brümmer | Application-independent evaluation of speaker detection[END_REF]. 10 How good are two classes separated for any threshold? 11 In the light of domain shifts (e.g., changing speech signal quality between training and evaluation sets), scores loose the capability of correctly encoding the expected class ratio for what their value suggests; scores are not LLRs anymore and thus loose their goodness for ideal decision making. The calibration loss quantifies this gap; scores are considered to be LLRs if C llr ≤ 1, and for C llr = 1, a coin toss provides the same information.

to imagine a scenario in which the anonymized sample is from an incoming telephone call, and the subjects need to judge whether the voice in the call is similar to the clean voice of the claimed speaker. The instruction given to the subjects is as follows:

Imagine you are working for a bank call center. Your task is to compare customer inquiries with voices recorded when the same customer made inquiries in the past. You must determine whether the voices are of the same person or another person who is impersonating the original voices. However, if you choose 'spoofing by someone else' more than necessary, there will be many complaints from real customers, which should be avoided. Imagine a situation in which you are working to protect bank accounts and balance convenience. Now press the 'Sample A' and 'Sample B' buttons below and listen to the samples. You can listen to them as many times as you like. Use only the audio you hear to determine if the speakers are the same or not. The content of the conversation in English is irrelevant and does not need to be heard. Please judge on the basis of the characteristics of the voice, not the content of the words.

As the instruction describes, the subject will listen to two samples (i.e., Sample A and Sample B) in one evaluation page. Sample B is always a randomly selected enrollment utterance of a given original speaker. Sample A may be an anonymized trial utterance or a clean trial utterance of the same original speaker or a different speaker. The subject will be asked to evaluate the similarity using a scale of 1 to 10, where 1 denotes 'different speakers' and 10 denotes 'the same speaker' with highest confidence. Note that, by using clean speech of the same original speaker or a different speaker as Sample A, we will have anchors in the listening test and can visualize the performance of each participant system through Detection Error Tradeoff (DET) curves.

Subjective speaker linkability

The second subjective evaluation metric is based on speaker linkage, i.e., clustering. The listeners will be asked to place a set of anonymized trial utterances from different speakers in a 1-or 2dimensional space according to speaker similarity. This will be done using a specially developed graphical interface, where each utterance is represented by a point in the space. The distance between two points expresses the subjective perception of speaker dissimilarity between the corresponding utterances.

Subjective speech intelligibility

Naturalness of the anonymized speech will also be evaluated through a large-scale crowdsourced evaluation. For one subjective evaluation round, the subject will listen to a sample and evaluate its naturalness using a scale from 1 (completely unintelligible) to 10 (completely intelligible and all content is clear). The sample can be an anonymized trial utterance or a clean enrollment utterance from a randomly selected speaker. The results can be visualized through DET curves for comprehensive comparison.

Subjective speech naturalness

Similar to the subjective speech intelligibility evaluation, naturalness of the anonymized speech will also be evaluated through a large-scale crowdsourced evaluation. The subject will listen to a sample and evaluate its naturalness using a scale from 1 (completely unnatural) to 10 (completely natural). The sample can be an anonymized trial utterance or a clean enrollment utterance from a randomly selected speaker. The performance of the participants can be visualized through DET curves for comprehensive comparison.

Baseline

Two different baseline systems have been developed for the challenge 12 : (1) anonymization using x-vectors and neural waveform models, and (2) anonymization using McAdams coefficient.

Baseline-1: Anonymization using x-vectors and neural waveform models

Our primary baseline system is based on the speaker anonymization method proposed in [START_REF] Fang | Speaker anonymization using X-vector and neural waveform models[END_REF] and shown in Figure 3. The anonymization is performed in three steps:

• Step 1: Feature extraction: extract the speaker x-vector [START_REF] Snyder | X-vectors: Robust DNN embeddings for speaker recognition[END_REF], the fundamental frequency (F0) and bottleneck (BN) features from the original audio waveform.

• Step 2: X-vector anonymization: anonymize the x-vector of the source speaker using an external pool of speakers.

• Step 3: Speech synthesis: synthesize the speech waveform from the anonymized x-vector and the original BN and F0 features using an acoustic model and a neural waveform model. In order to implement these steps, four different models are required as shown in Figure 3. Details for training these components are presented in Table 7.

In Step 1, to extract BN features, an ASR acoustic model (AM) is trained (#1 in Table 1). We assume that these BN features represent the linguistic content of the speech signal. The ASR AM has a factorized time delay neural network (TDNN-F) model architecture [START_REF] Povey | Semiorthogonal low-rank matrix factorization for deep neural networks[END_REF][START_REF] Peddinti | A time delay neural network architecture for efficient modeling of long temporal contexts[END_REF] and is trained using the Kaldi toolkit [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF]. To encode speaker information, an x-vector extractor with a TDNN model topology (#2 in Table 1) is also trained using Kaldi.

In Step 2, for a given source speaker, a new anonymized x-vector is computed by averaging a set of candidate x-vectors from the speaker pool for which the similarity to the x-vector of the source speaker is in the given range. The cosine distance cos (x 1 , x 2 ) or, optionally, probabilistic linear discriminant analysis (PLDA)) distance is used as a similarity measure between two x-vectors x 1 and x 2 . The candidate x-vectors for averaging are chosen in two steps. First, for a given x-vector, N the most farthest candidates from the speaker pool are selected. Second, a smaller subset of N * x-vector candidates from this set are chosen randomly 13 . The x-vectors for the speaker pool are extracted from a disjoint dataset (LibriTTS-train-other-500 ).

In Step 3, two modules are used to generate the speech waveform: a speech synthesis AM that generates Mel-filterbank features given the F0, the anonymized x-vector, and the BN features, and a neural source-filter (NSF) waveform model [START_REF] Wang | Neural harmonic-plus-noise waveform model with trainable maximum voice frequency for text-to-speech synthesis[END_REF] that produces a speech waveform given the F0, the anonymized x-vector, and the generated Mel-filterbanks. Both models (#3 and #4 in Table 1) are trained on the same corpus (LibriTTS-train-clean-100 ).

More details about the baseline recipe can be found in the provided scripts.

Results for the ASV objective evaluation are provided in Table 8 for the development and evaluation datasets. Results for ASR evaluation are presented in Table 9 in terms of WER.

Baseline-2: Anonymization using McAdams coefficient

A secondary, alternative baseline, in contrast to the primary baseline, does not require any training data and is based upon simple signal processing techniques. It employs the McAdams coefficient [START_REF] Mcadams | Spectral fusion, spectral parsing and the formation of the auditory image[END_REF][START_REF] Patino | Speaker anonymisation using the McAdams coefficient[END_REF] to achieve anonymisation by shifting the pole positions derived from linear predictive coding (LPC) analysis of speech signals. The process is depicted in Figure 4. It starts with the application of frame-by-frame LPC sourcefilter analysis to derive the LPC coefficients and residuals for a speech signal. Residuals are set aside and retained for later resynthesis. LPC coefficients are converted into a set of pole positions. The McAdams transformation is then applied to the angles of each pole (with respect to the origin in the z-plane), each one of which corresponds to a peak in the spectrum (loosely resembling formant positions). While real-valued poles are left unmodified, the angles ϕ of the poles with a non-zero imaginary part (with values between 0 and π radians) are raised to the power of the McAdams coefficient α so that a transformed pole has new angle ϕ α . For angles ϕ < 1 (in radians) and α > 1, ϕ α results in a negative shift in angle, whereas for α < 1, ϕ α results in a positive shift. For ϕ > 1, the shift is positive for α > 1 and negative for α < 1. The effect of such manipulation upon a of arbitrary pole positions is illustrated in Figure 6 for values of α = {0.9, 1.1}. The operation results in the contraction or expansion of the pole positions around ϕ = 1. The effect upon the corresponding magnitude spectrum is illustrated in Figure 5. For a sampling rate of 16kHz, i.e. for data used in the challenge, ϕ = 1 corresponds to approximately 2.5kHz which is the approximate mean formant position [START_REF] Ghorshi | Cross-entropic comparison of formants of british, australian and american english accents[END_REF]. Corresponding complex conjugate poles are similarly and shifted in the opposite direction to their counterparts.

The new, full set of poles, including original real-valued poles and shifted complex poles are then converted back to LPC coefficients. Finally, LPC coefficients and residuals are used to resynthesise a new speech frame in the time domain. Objective evaluation results for ASV and ASR are illustrated in Tables 10 andTable 11 respectively. This technique is similar in nature to the VoiceMask method [START_REF] Qian | Hidebehind: Enjoy voice input with voiceprint unclonability and anonymity[END_REF] that was applied to preserve privacy in speech [START_REF] Srivastava | Evaluating voice conversion-based privacy protection against informed attackers[END_REF]. While results are inferior to those for the primary baseline, it is stressed that the algorithm has not been optimised in any way.

Evaluation rules

• Participants are free to develop their own anonymization system, using parts of the baseline or not. They are also free to tune the compromise between speaker verifiability and speech quality/intelligibility according to their own preference.

• Participants can use only the training datasets and the enrollment datasets specified in Section 4.1 in order to train their system and tune its hyperparameters. Using additional speech data is not allowed. • Speaker anonymization must be done in a speaker-to-speaker manner. All enrollment utterances from a given speaker must be converted into the same pseudo-speaker, and enrollment utterances from different speakers must be converted into different pseudo-speakers. Also, the pseudo-speaker corresponding to a given speaker in the enrollment set must be different from the pseudo-speaker corresponding to that same speaker in the trial set.

• Modifications of ASV eval and ASR eval (such as changing decoder parameters) are not allowed.

• For every tested system, participants should report all objective metrics (WER, EER, C llr and C min llr ) on the two development datasets and the two evaluation datasets. Participants should compute these metrics themselves using the provided evaluation scripts. The organizers will be responsible for computing the subjective metrics and the post-evaluation metrics only.

• In the system description, participants should present their results in the same format as used for Tables 8 and9, or format used in the paper [START_REF] Tomashenko | Introducing the VoicePrivacy initiative[END_REF]. Check the following overleaf document for the allowed table formats to report the results: https://www.overleaf.com/read/ cdtgnppxsymj. In order to convert the result file generated by the evaluation system14 to L A T E X tables in the required format, participants can use the following script: https://github.com/ Voice-Privacy-Challenge/Voice-Privacy-Challenge-2020/blob/master/baseline/local/ results_to_latex.py.

Post-evaluation analysis

Following Deadline-2, the organizers will run additional evaluation experiments in order to further characterize the performance of the submitted systems and pave the way for the next Voice Privacy Challenge. To do so, we will ask volunteer participants to share with us the anonymized speech data obtained when running their anonymization system on the training/development datasets and on the evaluation dataset with different settings, and we will compute additional evaluation metrics on these data. The full details of the additional data to be provided by participants and the additional metrics which will be computed will be disclosed in due time. Roughly speaking, this involves:

1. retraining the evaluation systems (ASR eval and ASV eval ) on anonymized training data in order to evaluate the suitability of the proposed anonymization technique for ASR and ASV training [START_REF] Tomashenko | Post-evaluation analysis for the voice privacy 2020 challenge: Using anonymized speech data to train attack models and asr[END_REF];

2. computing subjective evaluation metrics (as described in Section 5.2) on selected subsets of evaluation data;

3. computing additional metrics to assess, e.g., the fulfillment of the goal that pseudo-speakers are sufficiently different from each other and the speaker verifiability performance in the presence of an attacker with additional knowledge.

9 Registration and submission of results

General mailing list

All participants and team members are encouraged to subscribe to the general mailing list. Subscription can be done by sending an email to: sympa@lists.voiceprivacychallenge.org with 'subscribe 2020' as the subject line. Successful subscriptions are confirmed by return email.

To post messages to the mailing list itself, emails should be addressed to: 2020@lists.voiceprivacychallenge.org

Registration

Participants/teams are requested to register for the evaluation. Registration should be performed once only for each participating entity and by sending an email to: organisers@lists.voiceprivacychallenge.org with 'VoicePrivacy 2020 registration' as the subject line. The mail body should include: (i) the name of the team; (ii) the name of the contact person; (iii) their country; (iv) their status (academic/nonacademic).

Submission of results

Each participant may make up to 5 different submissions/algorithms. In case of several system submissions, participants should indicate a primary system among them, and the rest systems should be marked as contrastive. Only primary systems will be considered in subjective evaluation. Each single submission should include:

1. Metric values for objective assessment (WER, EER, C llr and C min llr ) on the evaluation and development datasets according to the common protocol (participants should submit result file obtained by the provided evaluation scripts15 ).

Corresponding PLDA (LLR) scores in Kaldi format (for development and evaluation data)

obtained with the provided scripts;

3. Corresponding anonymized speech data (wav files, 16kHz, with the same names as in the original corpus) generated from the evaluation and development datasets. For evaluation, wav files will be converted to 16-bit Signed Integer PCM format, and this format is recommended for submission. These data will be used by the challenge organizers to verify the submitted scores, make post-evaluation analysis with other metrics and to run listening tests for subjective evaluation 16 .

All the anonymized speech data should be submitted in the form of a single compressed archive. Each participant should also submit a single, detailed system description. All submissions should be made according to the schedule below. Submissions received after the deadline will be marked as 'late' submissions, without exception. System descriptions will be will be made publicly available on the Challenge website. Further details concerning the submission procedure and URI will be published via the participants mailing list and via the VoicePrivacy Challenge website.

10 Special session at Interspeech 2020 A special session on the VoicePrivacy 2020 Challenge will be held at Interspeech 2020 in Shanghai, China.

All participants are encouraged (but not obliged) to submit their papers, dedicated to early version of their challenge entry, to the special session of the conference by 8th May 2020 (which corresponds to the Deadline-1 in the Challenge schedule, see Section 11). The review process for papers submitted to the special session is the same as for regular papers. Scientific novelty of the proposed ideas should be clearly stated and evaluated experimentally. Subjective evaluation results will be obtained only after Deadline-2 and can be included (optionally) in the camera-ready version of the paper. Other participants may submit papers to the challenge only.

Accepted Interspeech papers and other Challenge submissions will both be presented at the special session in the form of posters. These two types of presentations will be clearly differentiated in the program.

Schedule

There will be two submission deadlines (early Deadline-1 and late Deadline-2) in the Challenge. Participants may submit to one or both deadlines. Interspeech paper submission is encouraged but optional. All participants will be invited to present their work at the VoicePrivacy session/event.

12 Acknowledgement

1 1 2 Figure 1 :

 121 Figure 1: ASR evaluation for (1) clean speech data and (2) anonymized speech data. WER is estimated on the trial utterances of the development and evaluation datasets.

1 . 3 .

 13 Compute PLDA (LLR) scores for (a) clean enrollment data and (b) anonymized trial data; 2. Compute PLDA (LLR) scores for (a) anonymized enrollment data and (b) anonymized trial data; For steps 1 and 2, estimate the corresponding speaker verifiability metrics: equal error rate (EER) and log-likelihood-ratio cost function (C llr ).

Figure 2 :

 2 Figure 2: ASV evaluation for (1) clean trial and enrollment data; (2) anomymized trial data and clean enrollment data; and (3) anomymized trial and enrollment data.

Figure 3 :

 3 Figure 3: Baseline system.

Figure 4 :Figure 5 :

 45 Figure 4: Pipeline of the application of the proposed McAdams coefficient-based approach to anonymisation on a speech frame basis. The angle ϕ of poles with a non-zero imaginary part are raised to the power of the McAdams coefficient α to provoke a shift in frequency of its associated formant.

Figure 6 :

 6 Figure 6: Example of pole-zero map as shown in Figure 5

  This work was supported in part by the French National Research Agency under projects HAR-POCRATES (ANR-19-DATA-0008) and DEEP-PRIVACY (ANR-18-CE23-0018), by the European Union's Horizon 2020 Research and Innovation Program under Grant Agreement No. 825081 COM-PRISE (https://www.compriseh2020.eu/), and jointly by the French National Research Agency and the Japan Science and Technology Agency under project VoicePersonae. The authors would like to thank Md Sahidullah and Fuming Fang.

Table 1 :

 1 Statistics of the training datasets.

	Subset	Size,h	Number of Speakers Female Male Total	Number of Utterances
	VoxCeleb-1,2	2,794	2,912 4,451	7,363	1,281,762
	LibriSpeech: train-clean-100	100	125	126	251	28,539
	LibriSpeech: train-other-500	497	564	602	1,166	148,688
	LibriTTS: train-clean-100	54	123	124	247	33,236
	LibriTTS: train-other-500	310	560	600	1,160	205,044

Table 2 :

 2 Statistics of the development datasets.

	Subset

Table 3 :

 3 Statistics of the evaluation datasets.

	Subset

Table 4 :

 4 Statistics of the training dataset for the ASV eval and ASR eval evaluation systems.

	Subset	Size,h	Number of Speakers Female Male Total	Number of Utterances
	LibriSpeech: train-clean-360	363.6	439	482	921	104,014

Table 5 :

 5 Number of trials in the development datasets.

	Subset	Trials	Female	Male Total
	Librispeech: dev-clean	Target Impostor	704 14,566 12,796 27,362 644 1,348
		Target (common part)	344	351	695
	VCTK-dev	Target (different part) Impostor (common part)	1,781 4,810	2,015 4,911	3,796 9,721
		Impostor (different part)	13,219 12,985 26,204

Table 6 :

 6 Number of trials in the evaluation datasets.The EER is computed from the PLDA scores as follows. Let P fa (θ) and P miss (θ) denote the false alarm and miss rates at threshold θ: so that P fa (θ) and P miss (θ) are, respectively, monotonically decreasing and increasing functions of θ. EER corresponds to the threshold θ EER at which the two detection error rates are equal, i.e., EER = P fa (θ EER ) = P miss (θ EER ).

	Subset	Trials	Female	Male Total
	Librispeech: test-clean	Target Impostor	548 11,196	449 9,457 20,653 997
		Target (common part)	346	354	700
	VCTK-test	Target (different part) Impostor (common part)	1,944 4,838	1,742 4,952	3,686 9,790
		Impostor (different part)	13,056 13,258 26,314

Table 7 :

 7 Baseline anonymization system: models and corpora. The model indexes are the same as in Figure3. Superscript numbers represent feature dimensions.

	# Model Description	Output features	Training dataset
			TDNN-F
			Input: MFCC 40 + i-vectors 100	BN 256 features	Librispeech:
	1 ASR AM	17 TDNN-F hidden layers	extracted from	train-clean-100
			Output: 6032 triphone ids	the final hidden layer	train-other-500
			LF-MMI and CE criteria
			TDNN
	2	X-vector extractor	Input: MFCC 30 7 hidden layers + 1 stats pooling layer Output: 7232 speaker ids	speaker x-vectors 512	VoxCeleb: 1, 2
			CE criterion
			Autoregressive (AR) network
	3	Speech synthesis AM	Input: F0 1 + BN 256 +x-vectors 512 FF * 2 + BLSTM + AR + LSTM * 2 + highway-postnet	Mel-filterbanks 80	LibriTTS: train-clean-100
			MSE criterion
	4	NSF model	sinc1-h-NSF in [15] Input: F0 1 + Mel-fbanks 80 + x-vectors 512 STFT criterion	speech waveform	LibriTTS: train-clean-100
			Pool of speaker x-vectors		LibriTTS: train-other-500

Table 8 :

 8 ASV results for Baseline-1 for development and test data (o -original, a -anonymized speech data for enrollment (Enr) and trial (Trl) parts; Gen denotes speaker gender: f -female, m -male).

	# Dev. set	EER,% C min llr	Cllr Enr Trl Gen Test set	EER,% C min llr	Cllr
	1 libri_dev	8.665 0.304 42.857	o	o	f	libri_test	7.664 0.183 26.793
	2 libri_dev	50.140 0.996 144.112	o	a	f	libri_test	47.260 0.995 151.822
	3 libri_dev	36.790 0.894 16.345	a	a	f	libri_test	32.120 0.839 16.270
	4 libri_dev	1.242 0.034 14.250	o	o	m	libri_test	1.114 0.041 15.303
	5 libri_dev	57.760 0.999 168.988	o	a	m	libri_test	52.120 0.999 166.658
	6 libri_dev	34.160 0.867 24.715	a	a	m	libri_test	36.750 0.903 33.928
	7 vctk_dev_com	2.616 0.088	0.868	o	o	f	vctk_test_com	2.890 0.091	0.866
	8 vctk_dev_com	49.710 0.995 172.049	o	a	f	vctk_test_com	48.270 0.994 162.531
	9 vctk_dev_com	27.910 0.741	7.205	a	a	f	vctk_test_com	31.210 0.830	9.015
	10 vctk_dev_com	1.425 0.050	1.559	o	o	m	vctk_test_com	1.130 0.036	1.041
	11 vctk_dev_com	54.990 0.999 192.924	o	a	m	vctk_test_com	53.390 1.000 190.136
	12 vctk_dev_com	33.330 0.840 23.891	a	a	m	vctk_test_com	31.070 0.835 21.680
	13 vctk_dev_dif	2.864 0.100	1.134	o	o	f	vctk_test_dif	4.887 0.169	1.495
	14 vctk_dev_dif	49.970 0.989 166.027	o	a	f	vctk_test_dif	48.050 0.998 146.929
	15 vctk_dev_dif	26.110 0.760	8.414	a	a	f	vctk_test_dif	31.740 0.847 11.527
	16 vctk_dev_dif	1.439 0.052	1.158	o	o	m	vctk_test_dif	2.067 0.072	1.817
	17 vctk_dev_dif	53.950 1.000 167.511	o	a	m	vctk_test_dif	53.850 1.000 167.824
	18 vctk_dev_dif	30.920 0.839 23.797	a	a	m	vctk_test_dif	30.940 0.834 23.842

Table 9 :

 9 ASR results for Baseline-1 in terms of WER,% for development and test data (o-original, a-anonymized speech) for two trigram language models (LMs): LM s -small, and LM l -large LM.

	#		Dev set	WER, % LMs LM l	Data	Test set	WER, % LMs LM l
	1	libri_dev	5.25	3.83	o	libri_test	5.55	4.15
	2	libri_dev	8.76	6.39	a	libri_test	9.15	6.73
	3	vctk_dev 14.00 10.79	o	vctk_test 16.39 12.82
	4	vctk_dev 18.92 15.38	a	vctk_test 18.88 15.23
	Speech frame	LPC analysis
			Residuals

Table 10 :

 10 ASV results for Baseline-2 for development and test data (o -original, a -anonymized speech data).

	# Dev. set	EER,% C min llr	Cllr Enr Gen Test set	EER,% C min llr	Cllr
	1 libri_dev	8.807 0.305 42.903	o	o	f	libri_test	7.664 0.184 26.808
	2 libri_dev	35.370 0.821 116.892	o	a	f	libri_test	26.090 0.685 115.571
	3 libri_dev	23.440 0.621 11.726	a	a	f	libri_test	15.330 0.490 12.553
	4 libri_dev	1.242 0.035 14.294	o	o	m	libri_test	1.114 0.041 15.342
	5 libri_dev	17.860 0.526 105.715	o	a	m	libri_test	17.820 0.498 106.434
	6 libri_dev	10.560 0.359 11.951	a	a	m	libri_test	8.241 0.263 15.376
	7 vctk_dev_com	2.616 0.088	0.869	o	o	f	vctk_test_com	2.890 0.092	0.861
	8 vctk_dev_com	34.300 0.877 85.902	o	a	f	vctk_test_com	30.640 0.807 93.967
	9 vctk_dev_com	11.630 0.366 43.551	a	a	f	vctk_test_com	14.450 0.465 42.734
	10 vctk_dev_com	1.425 0.050	1.555	o	o	m	vctk_test_com	1.130 0.036	1.042
	11 vctk_dev_com	23.930 0.669 90.757	o	a	m	vctk_test_com	24.290 0.713 99.336
	12 vctk_dev_com	10.540 0.316 24.986	a	a	m	vctk_test_com	11.860 0.349 28.225
	13 vctk_dev_dif	2.920 0.101	1.135	o	o	f	vctk_test_dif	4.938 0.169	1.492
	14 vctk_dev_dif	35.540 0.907 90.540	o	a	f	vctk_test_dif	30.040 0.794 93.211
	15 vctk_dev_dif	15.830 0.503 39.811	a	a	f	vctk_test_dif	16.920 0.546 41.341
	16 vctk_dev_dif	1.439 0.052	1.155	o	o	m	vctk_test_dif	2.067 0.072	1.816
	17 vctk_dev_dif	28.240 0.741 98.419	o	a	m	vctk_test_dif	28.240 0.720 101.704
	18 vctk_dev_dif	11.220 0.384 23.093	a	a	m	vctk_test_dif	12.230 0.397 25.064

Table 11 :

 11 ASR results for Baseline-2 in terms of WER,% for development and test data (o-original, a-anonymized speech) for two trigram LMs: LM s -small, and LM l -large LM.

https://www.spsc-sig.org

Librispeech: http://www.openslr.org/12

LibriTTS: http://www.openslr.org/60/

VCTK, release version 0.92: https://datashare.is.ed.ac.uk/handle/10283/3443

VoxCeleb: http://www.openslr.org/60/

Elicitation paragraph: http://accent.gmu.edu/pdfs/elicitation.pdf

Rainbow passage: https://www.dialectsarchive.com/the-rainbow-passage

Evaluation scripts: https://github.com/Voice-Privacy-Challenge/Voice-Privacy-Challenge-2020

In the baseline, the following parameter values are used: N = 200 and N * = 100; and PLDA was used as the distance between x-vectors.

Example: https://github.com/Voice-Privacy-Challenge/Voice-Privacy-Challenge-2020/blob/master/ baseline/RESULTS_baseline

Example of the result file for the baseline system: https://github.com/Voice-Privacy-Challenge/ Voice-Privacy-Challenge-2020/blob/master/baseline/RESULTS_baseline

Please note that subjective evaluation will be performed for Deadline-2 submissions only, not for Deadline-1