Mohammad Mohammadamini
email: mohammad.mohammadamini@univ-avignon.fr

Nathan Griot
email: nathan.griot@univ-avignon.fr

Driss Matrouf
email: driss.matrouf@univ-avignon.fr

Learning linguistic content embeddings for phrase and language independent utterance verification

Keywords: utterance verification, linguistic-content embeddings, text-dependent speaker verification, keyword spotting

In this paper, we introduce a phrase and language-independent utterance verification system. The objective of an utterance verification system is to confirm whether the linguistic content of two speech utterances is the same or not. Our proposed approach is phrase and language-independent, therefore it can generalize for unseen phrases and even unseen languages. The proposed framework is based on a ResNet embedding extractor trained on the Common Voice dataset which is optimized by a linguistic content classification task.The proposed approach can be used in text-dependent speaker recognition systems, passphrase verification systems, and keyword spotting systems. Our system is tested on several protocols including Deepmine for text-dependent speaker verification and Speech Commands' keyword-spotting benchmarks. Obtaining high performance for unseen phrases and languages makes our approach plausible for utterance verification in low-resource and even zero-resource languages. For example, the EER for a protocol on Common Voice English is 0.16 while for Common Voices French without using French training data the EER is 1.23.

Introduction

A phrase recognizer or an utterance verification system confirms whether two utterances contain the same linguistic content or not [START_REF] Boulianne | Language-independent voice passphrase verification[END_REF][START_REF] Zeinali | Spoken Pass-Phrase Verification in the i-vector Space[END_REF]. Utterance verification systems can be utilized in various speech technologies such as text-dependent speaker recognition systems, passphrase verification systems, command systems, and keyword spotting systems. A speech signal encodes different kinds of information such as speaker, language, emotion, and linguistic content. Obtaining disentangled representations that encode a particular aspect of information is used in different speech technologies such as speaker recognition [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF], language identification [START_REF] Snyder | Spoken Language Recognition using X-vectors[END_REF], and emotion recognition [START_REF] Pappagari | X-vectors meet emotions: A study on dependencies between emotion and speaker recognition[END_REF]. In this paper, we propose a new approach for utterance verification that is based on fixed-length embeddings that capture linguistic content.

The previous work on phrase/utterance verification is mainly limited to a close set of sentences [START_REF] Boulianne | Language-independent voice passphrase verification[END_REF][START_REF] Zeinali | Spoken Pass-Phrase Verification in the i-vector Space[END_REF]. In a closeset setup, the verification of new utterances requires preparing new data and system modification which leads to less flexibility, less reusability, and more time and money consumption. In this paper, we introduce a phrase and language-independent utterance verification system that extends the previous work in several ways. The proposed system makes it possible to generate an embedding characterizing the linguistic content regardless of the sentence pronounced, even when it has never been seen in the training data. From this perspective, we call the proposed system phrase-independent. Furthermore, we will show that with a relatively small performance degradation, it can be used even for new languages without training data for that language. In this sense, we call our approach language-independent.

A common application for an utterance verification system is speaker recognition systems. Utterance verification is used in text-dependent speaker recognition systems in two ways. Firstly it can be used to filter out the trials containing different linguistic content. By doing so, the linguistic variability can be controlled which leads to higher performance in severe acoustical conditions [START_REF] Larcher | Text-dependent speaker verification: Classifiers, databases and rsr2015[END_REF][START_REF] Tomi Kinnunen | Utterance Verification for Text-Dependent Speaker Recognition: A Comparative Assessment Using the RedDots Corpus[END_REF]. Also, passphrase verification can be used as a secondary metric alongside the speaker characteristics to make the speaker recognition systems more robust [START_REF] Boulianne | Language-independent voice passphrase verification[END_REF]. The previous work in this direction is mainly focused on passphrase verification for a small pool of sentences. The RSR2015 [START_REF] Larcher | Text-dependent speaker verification: Classifiers, databases and rsr2015[END_REF] contains 30 unique sentences in the training part; Deepmine is another well-known dataset that contains 5 English and 5 Persian phrases for text-dependent speaker recognition [START_REF] Zeinali | DeepMine Speech Processing Database: Text-Dependent and Independent Speaker Verification and Speech Recognition in Persian and English[END_REF]. Our proposed approach can exempt us from the time and moneyconsuming tasks of preparing text-dependent training datasets.

Keyword spotting systems and command systems are becoming more popular in speech assistants. The goal of such systems is to recognize specific content in an audio stream to process a command [START_REF] Pez-Espejo | Deep spoken keyword spotting: An overview[END_REF]. The main configuration of keyword spotting systems is based on classifying a close set of words/phrases [START_REF] Pez-Espejo | Deep spoken keyword spotting: An overview[END_REF]. Similar to text-dependent speaker recognition systems the benchmarks are limited to a small number of sentences or words [START_REF] Warden | Speech commands: A dataset for limited-vocabulary speech recognition[END_REF]. For example, Speech Commands is a known keyword spotting benchmark that contains 35 unique words commonly used as commands in speech assistants. Also, the majority of datasets are devoted to high-resource languages such as English or Mandarin [START_REF] Pez-Espejo | Deep spoken keyword spotting: An overview[END_REF]. The fact that our approach is phraseindependent and language-independent makes it a more flexible and reusable keyword-spotting system. Indeed, the users can define their commands for voice assistants without having the written or transcribed data in their language. Also, it lets them personalize the commands for the voice assistants.

Wake word detection is a particular command recognition system that aims to activate a speech assistant by a specific command like "Hi Alexa" [START_REF] Tang | Howl: A deployed, open-source wake word detection system[END_REF][START_REF] Wang | Wake word detection with streaming transformers[END_REF]. In practice, such systems run on low-resource devices and listen continuously for a specific wake word. Similar to the aforementioned applications wake-word detection is constrained to recognizing a close set of phrases which makes the use of speech assistance more difficult for non-English speakers or underrepresented languages.

As it is described, the majority of available utterance verification systems are constrained to a small close set of phrases.

In this paper, we propose a general phrase and languageindependent utterance verification system by training discriminant linguistic content embeddings. In a similar approach to text-independent speaker recognition systems, we train a ResNet embedding extractor that captures discriminant linguistic content representations at the utterance level. To do so, it is required to have a transcribed/labeled speech dataset with multiple repetitions for each utterance. Driving a dataset for utterance verification from Common Voice is another contribution of this work. The details about the training dataset are presented in Section 3. To the best of our knowledge, this methodology is not previously used to achieve linguistic content representations for speech utterances.

The rest of the paper is organized as: in section 2 the configuration of the proposed utterance verification system and the architecture of the embedding extractor are described. In section 3 the training and evaluation datasets are presented, and section 4 discusses the obtained results.

Utterance verification system

Different kinds of information such as speaker characteristics, recording devices, recording environment, and linguistic content are encoded in a speech signal. Finding a compact representation for a variable-length speech signal for different tasks such as speaker recognition [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF], language identification [START_REF] Snyder | Spoken Language Recognition using X-vectors[END_REF], and emotion recognition [START_REF] Pappagari | X-vectors meet emotions: A study on dependencies between emotion and speaker recognition[END_REF] is explored. In this paper, we propose an approach to learn representations that captures linguistically discriminant characteristics. The idea of the proposed system is taken from speaker embedding extractions [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF][START_REF] Bai | Speaker recognition based on deep learning: An overview[END_REF][START_REF] Xue | Ecapa-tdnn for multi-speaker textto-speech synthesis[END_REF]. Our proposed system is depicted in Figure 1. The training and application parts of the proposed configuration are described in the following.

System configuration

Training phase

The training phase is composed of three steps. Firstly the acoustic features are extracted. After that, we have an utterance embedding extractor. In our case, we used ResNet34 which is described in Table 1. Finally, there is an utterance classification that classifies the utterances based on their linguistic content. By imposing on the output of the embedding extractor to generate linguistically discriminant representations, the embedding extractor captures the linguistic content information.

Application phase

In the application phase, we use the trained utterance embedding extractor to generate a linguistic content discriminant representation. The obtained representations are expected to encode the linguistic content characteristics independent of their language and spoken phrase. In this step, the classifier is removed and the given representations by the trained embedding extractor are used for utterance verification. For a pair of given embeddings, a decision on their sameness in terms of content will be taken based on the similarity between the embeddings. We are using the cosine distance to compare each pair of representations, if this distance is smaller than a threshold the embeddings are considered to have the same linguistic content, otherwise, they will be considered to have different content.

As we mentioned in Section 1, the proposed utterance verification can be used for several applications including passphrase verification, text-dependent speaker recognition, command recognition, and word wake detection.

• Passphrase verification: Authenticating the user's linguistic content as a secondary metric alongside the speaker characteristics can improve the reliability of speaker recognition systems. In this application for each registered user, both speaker characteristics and spoken phrases (passphrases) should be authenticated. Since our proposed approach is phrase-independent, the users can change their passphrase. In this configuration, if the passphrase is not revealed the speaker recognition system remains safe facing some spoofing attacks including the replaying attack, voice conversion, and personification attacks [START_REF] Wang | Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech[END_REF].

• Text-dependent speaker verification:

In textdependent speaker recognition systems, only specific phrases are accepted that lead to reducing the linguistic variability and higher performance of speaker recognition systems [START_REF] Larcher | Text-dependent speaker verification: Classifiers, databases and rsr2015[END_REF]. Our utterance verification systems can be used as a preprocessing step to filter out the undefined phrases in the system.

• Command recognition: In this case, there is a set of registered commands [START_REF] Warden | Speech commands: A dataset for limited-vocabulary speech recognition[END_REF]. Our proposed approach can support both open and closed set commands. Therefore, the user can add or customize the voice commands. The generalizability of the proposed approach to new languages without having training data can make the speech assistants more user-friendly and available for a bigger community.

• Word wake detection: In this case there is normally one registered command that wakes up the speech assistant. This configuration is a special case of a command recognition system. Having a phrase-independent utterance verification system, the user can define a personalized wake word for the speech assistants.

Speaker embedding extractor

The embedding extractor used in this paper is a variant based on ResNet [START_REF] Bai | Speaker recognition based on deep learning: An overview[END_REF]. The ResNet model for extracting embeddings consists of three modules: a set of ResNet Blocks, a statisticslevel layer, and segment-level representation layers.

• ResNet (Residual Network) uses stacks of many Residual Blocks. A Residual Block is made up of two 2-twodimensional convolutional Neural Network (CNN) layers separated by a non-linearity (ReLU). The input of the residual block is added to its output to constitute the input of the next block.

• The statistics-level component is an essential component in converting a variable-length speech signal into a single fixed-dimensional vector. We are using the attentive statistics pooling [START_REF] Okabe | Attentive Statistics Pooling for Deep Speaker Embedding[END_REF], which aggregates over frame-level output vectors of the DNN and computes a weighted mean and weighted standard deviation.

• The segment-level component maps the segment-level vector to the utterance class. The weighted mean and weighted standard deviation are concatenated together and forwarded to the next layers and finally to the softmax output layer.

The detailed topology of the used ResNet is shown in Table 1. Batch-norm and ReLU layers are not shown. The dimensions are (Frequency×Channels×Time). The input is comprised of

LArcF ace = - 1 n n i=1 log e s • (cos θy i + m)
e s. cos (θy i + m) + j̸ =y i e s.(cos θ j ) (1) Where yi is the ith utterance, s is a scale factor and m is the margin.

Dataset

Training data

The majority of utterance-verification research is done on limited task-specific datasets such as text-dependent speaker recognition [START_REF] Zeinali | Spoken Pass-Phrase Verification in the i-vector Space[END_REF][START_REF] Liu | Speaker-Utterance Dual Attention for Speaker and Utterance Verification[END_REF][START_REF] Liu | Phoneme-Aware and Channel-Wise Attentive Learning for Text Dependent Speaker Verification[END_REF][START_REF] Zeinali | SdSV Challenge 2020: Large-Scale Evaluation of Short-Duration Speaker Verification[END_REF]. The RSR2015 [START_REF] Larcher | Text-dependent speaker verification: Classifiers, databases and rsr2015[END_REF] and Deepmine [START_REF] Zeinali | DeepMine Speech Processing Database: Text-Dependent and Independent Speaker Verification and Speech Recognition in Persian and English[END_REF] are two common datasets used in the literature. In this section, we introduce a new dataset driven from Common Voice 15 to train a general-purpose phrase and language-independent utterance verification system.

Training the proposed linguistic content embedding extractor needs a big transcribed/labeled speech dataset with several repetitions per sentence. Common Voice is among the rare speech recognition datasets that have this specification partially. In our research, we drive the training dataset from the English part of Common Voice 15 1 . To achieve a relatively balanced training dataset we chose sentences that have more than 5 and less than 100 repetitions. The resulting dataset comprises 15,455 unique sentences. The total number of utterances is about 505k and 416 hours of speech data. The training data specifications are presented in table 2.

Evaluation data

We used several evaluation datasets that are described in this subsection. In evaluation protocols, target trials are pairs of utterances with the same linguistic content, and nontarget trials are pairs with different linguistic content.

• Common Voice English: In this protocol, 1740 unique utterances with 4 or 5 repetitions for each phrase from the English part of Common Voice 15 are chosen. There is no overlap between the chosen files and the training part described in section 3 in terms of linguistic content. From these files, 773k trials are generated.

• Common Voice French: In this protocol 225 unique files with 10 repetitions from the French part of Common Voice 15 are chosen. In this protocol, there are 381k trials. The goal of this protocol is to show the generalization of the proposed system for new languages.

• Deepmine English: Deepmine is a dataset for textdependent speaker recognition. In this protocol, there are 27k trials generated from the recording of 5 unique phrases in the Deepmine dataset. This protocol is a case study of our approach to text-dependent speaker recognition [START_REF] Zeinali | DeepMine Speech Processing Database: Text-Dependent and Independent Speaker Verification and Speech Recognition in Persian and English[END_REF].

• Deepmine Persian: In this protocol there are 25.4k trials generated from the Persian part of the Deepmine corpus which includes different recordings for 5 unique utterances [START_REF] Zeinali | DeepMine Speech Processing Database: Text-Dependent and Independent Speaker Verification and Speech Recognition in Persian and English[END_REF]. This protocol shows the generalizability to another new language.

• Speech Commands: Speech Commands is a widely used dataset for keyword spotting recognition. In this protocol, each utterance contains a single word, and the total number of unique utterances is 35. This protocol is created from 2000 randomly chosen files from the evaluation part of the Speech Commands dataset [START_REF] Warden | Speech commands: A dataset for limited-vocabulary speech recognition[END_REF].

Table 3 summarizes the details of evaluation protocols. The last columns show the average duration of test files in seconds.

Results and Discussion

The utterance embedding extractor is trained with driven data from Common Voice 15 discussed in section 3. For each clean file, four augmented versions are created which leads to 2.5m samples in the training data. The training data is augmented with different branches of Musan [START_REF] Sell | Musan: A music, speech, and noise corpus[END_REF] and RIR files [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF] using the Kaldi toolkit [START_REF] Boulianne | The kaldi speech recognition toolkit[END_REF]. The embedding extractor is optimized in 2, 000 iterations, with a learning rate started by 0.2 with a decay rate of 10 -4 . The size of each mini-batch is 128 and the mini-batch files are chosen randomly.

In the test step, the Equal Error Rate (EER) is calculated based on the cosine distance between pairs of files. If the cosine distance is lower than a threshold the files will be considered as same otherwise they have different linguistic content.

Common Voice protocols

As it is presented in Table 4, the EER is 0.16 on the Common Voice English protocol. The results show a high performance of the proposed approach. In the second experiment on Common Voice French, it is shown that extracted embeddings capture discriminant language characteristics regardless of the language. In this experiment, the EER is 1.2. However, there is relative performance degradation, but even without fine-tuning, it is a plausible result.

Utterance verification for text-dependent speaker recognition

In the second group of experiments; we did utterance verification on the Deepmine corpus which is a text-dependent speaker recognition. Similar to Common Voice protocols, the EER for the English version is 0.16, and for the Persian part, it is 1.9 (Table 4). Since, the Persian language and English versions are very different in terms of lexicon, syntax, and morphology, the obtained results prove the feasibility of repeating our approach for other new languages. Our last experiment is devoted to a command verification protocol created on Speech Commands. In this experiment, the EER increased is 6.0. The obtained result on this dataset is competitive in comparison to reported results on the same dataset [START_REF] Warden | Speech commands: A dataset for limited-vocabulary speech recognition[END_REF].

To have a more precise interpretation of the errors in this protocol we observed that the majority of errors come from trials where the only phonological difference between two files is one consonant. For example, there are pairs of "three" and "tree" or "on", and "off" trials which are the main resource of the error.

Robustness to duration

In other domains such as speaker recognition, the weakness of embeddings for short-duration utterances is significant [START_REF] Zeinali | Short-duration speaker verification (sdsv) challenge 2021: the challenge evaluation plan[END_REF]. We see that in linguistic content embeddings, the duration doesn't impact the results significantly. In another experiment, the Common Voice En protocol is evaluated for test files with different durations. As shown in Table 5, the EER of short duration between [START_REF] Boulianne | Language-independent voice passphrase verification[END_REF][START_REF] Zeinali | Spoken Pass-Phrase Verification in the i-vector Space[END_REF] seconds is almost the same as files in the range of [START_REF] Snyder | Spoken Language Recognition using X-vectors[END_REF][START_REF] Pappagari | X-vectors meet emotions: A study on dependencies between emotion and speaker recognition[END_REF] seconds. 0.13 [START_REF] Snyder | X-vectors: Robust dnn embeddings for speaker recognition[END_REF][START_REF] Snyder | Spoken Language Recognition using X-vectors[END_REF] 0.12 [START_REF] Snyder | Spoken Language Recognition using X-vectors[END_REF][START_REF] Pappagari | X-vectors meet emotions: A study on dependencies between emotion and speaker recognition[END_REF] 0.16

Conclusion

In this paper, we proposed an utterance verification system based on discriminant linguistic content embeddings. The proposed system is phrase-independent and it can generalize to a high degree to new languages. Our approach is performing well in different tasks such as text-dependent speaker recognition and keyword spotting applications. This work can be extended in several ways. The same idea can be tested in the case of having several languages in the training data to capture a broader phonetic variability. Besides the English version of Common Voice 15, a part of Welsh, German, Persian, and Kabyle languages have several repetitions per sentence that can help to foster the research in this direction. Also replacing the font-end features with a self-supervised representation such as Wav2Vec [START_REF] Baevski | wav2vec 2.0: A framework for selfsupervised learning of speech representations[END_REF] or Whisper [START_REF] Radford | Robust speech recognition via large-scale weak supervision[END_REF] is expected to capture linguistic characteristics more explicitly.

Figure 1 :

 1 Figure 1: The configuration of proposed Phrase and language-independent utterance verification system

Table 1 :

 1 The baseline ResNet-34 architecture.The last row, Y is the number of unique utterances. The dimensions are (Frequency×Channels×Time). The input is comprised of 60 filter banks.

	Layer name	Structure		Output
	Input	-		60 × 400 × 1
	Conv2D-1	3 × 3, Stride 1	60 × 400 × 32
	ResNetBlock-1	3 × 3, 32 3 × 3, 32	× 3 , Stride 1	60 × 400 × 32
	ResNetBlock-2	3 × 3, 64 3 × 3, 64	× 4, Stride 2	30 × 200 × 64
	ResNetBlock-3	3 × 3, 128 3 × 3, 128	× 6, Stride 2	15 × 100 × 128
	ResNetBlock-4	3 × 3, 256 3 × 3, 256	× 3, Stride 2	8 × 50 × 256
	Flatten	-		8 × 2048
	Pooling	-		2048
	Dense1	-		256
	Dense2 (Softmax)	-		Y
	Total	-		-
	60 filter banks from speech segments. The ResNet is trained
	with the Additive Angular Margin Loss (ArCFace)[18] function

(Equation. 1)

. For n number of training examples and y number of unique utterances, the network is trained to minimize:

Table 2 :

 2 The Common Voice training data specifications for utterances with more than 5 repetitions per sentence.

	Item	N
	Unique utterances	15455
	Utterance repetitions	>5
	Total Utterances	≈ 505k
	Total Duration	416h

Table 3 :

 3 Evaluation protocols specifications.

	Protocol	Task	trials utterances duration(s)
	Common Voice EN General 773k	1740	3.25
	Common Voice Fr	General 381k	225	3.47
	Deepmine En	TDSV	27k	5	2.30
	Deepmine Fa	TDSV	25.4k	5	1.9
	Speech command	KWS	1m	35	0.73

Table 4 :

 4 Obtained results for evaluation protocols in EER.

	Protocol	EER
	Common Voice En 0.16
	Common Voice Fr	1.23
	Deepmine En	0.13
	Deepmine Fa	1.9
	Speech command	6.00
	4.0.3. Utterance verification for command verification

Table 5 :

 5 The impact of duration variability

	Duration(s) EER
	[1, 2]	0.15
	[2, 3]

https://commonvoice.mozilla.org/en/datasets