Binbin Xu

Tao Chongyang

Zidu 1+

Feng

Youssef Raqui
email: youssef.raqui@diappymed.com

Sylvie Ranwez

A Benchmarking on Cloud based Speech-To-Text Services for French Speech and Background Noise Effect

Keywords: Reconnaissance vocale, Evaluation comparative, Langue Française, Google Cloud Speech-To-Text, Microsoft Azure Cognitive Services, Amazon Transcribe, IBM Watson Speech-To-Text, Benchmarking, French language, Google Cloud, Microsoft Azure Cognitive Services, Amazon Transcribe, IBM Watson

Alors que les applications de reconnaissance vocale se sont imposées dans notre quotidien, il existe peu d'études à grande échelle pour comparer les performances des solutions de l'état de l'art. Ceci est d'autant plus vrai dans une langue autre que la langue anglaise. Cet article propose une telle analyse comparative basée sur 17 heures d'enregistrement en Français. Quatre systèmes sont analysés : Google Cloud Speech-To-Text, Microsoft Azure Cognitive Services, Amazon Transcribe, et IBM Watson Speech to Text. Chacun ayant été mis à l'épreuve de cinq niveaux de bruit de fond, c'est l'équivalent de 400 heures de discours qui sont analysées. Microsoft Azure Cognitive Services a montré les meilleurs résultats en terme de taux d'erreur et une bonne résistance au bruit, tandis que la sensibilité au bruit d'IBM Watson Speech to Text compromet son usage en situation réelle.

Introduction

Lots applications with automated speech recognition (ASR) or Speech-To-Text (STT) over the past few years have been developed to improve our daily life like personal voice assistant, or have been deeply integrated in many of business chains. Thanks to the substantial development of deep neural network (DNN), the performance of STT has been drastically improved. Like other deep neural network applications, today it is not surprising that in some situations, current STT can even outperform humans. The IBM/Appen human transcription study [START_REF] Saon | English conversational telephone speech recognition by humans and machines[END_REF] showed that word error rate of human parity is about 5.1%. Microsoft Research is the first team reaching this milestone. However, the outstanding performances in DNN is based on large amount of labeled training data. This is also the case for DNN models on STT. For languages other than English, there's much less high quality audio data like in English. In consequence, the performances of STT on other languages are in general lower than for English, especially for languages featuring rich morphology like French. Though many public Deep Neural Network models are available for offline use, retraining or regular updating require extensive computing power which prevents individuals or small business from accessing these models or using them in an efficient way. The choice will be the cloudbased API services. Actually, the most powerful STT systems are all cloud-based. Integrating these systems in an application or a product line requires at first a benchmarking on their performance. There exist many benchmarking studies on the performance of cloud-based STT services. However, they are often conducted with very small or small sample size, for example, 20-60 sentences or hundreds of sentences. The benchmarking on English from Picovoice is one of the few large scale tests on STT, which contains 2620 audio files (5h24m) from LibriSpeech dataset [START_REF] Picovoice | Speech-to-text benchmark[END_REF]. Benchmarking of cloud-based STT on French is even less stu-died. Another major negative factor on STT performance is the background noise. Very often, only clean speech record is processed. However, for most real-life application, the background noise can hardly be avoided. This should be taken into account in STT benchmarking as well. The objective of this study is to benchmark four most used Speech-To-Text API (Application Programming Interface) with a large French dataset : 6693 files, about 17hours speech record. Five levels of common background noise are added in the clean speech and evaluated additionally. In total, more than 400 hours speech are transcribed.

2 Speech-To-Text system and data [START_REF] Xiong | The microsoft 2017 conversational speech recognition system[END_REF]. As all the other STT systems, Microsoft's STT system is also integrated in the Cloud Computing platform. All the four STT services offer the possibility to customize (like domain-specific) or retrain the Speech-to-Text models.

However, since they are all black-boxed APIs, the background models and architectures are unknown, it is difficult to benchmark the customized models with different configurations in a fair way. So, only the basic models (APIs) are called.

Speech corpus

The basic audio dataset in this work is from WCE-SLT-LIG [START_REF] Besacier | Word confidence estimation for speech translation[END_REF][START_REF] Le | Joint asr and mt features for quality estimation in spoken language translation[END_REF]. This corpus contains 6693 speech utterances recorded by 42 native speakers.

Environmental noise corpus

In real world cases, most speech takes place in noisy environments. This is one of the main challenges in Speech-to-Text applications. To evaluate the effects on the STT quality, we introduce another recently released environmental noise dataset : Microsoft Scalable Noisy Speech Dataset (MS-SNSD) [START_REF] Reddy | A scalable noisy speech dataset and online subjective test framework[END_REF]. The dataset provides a variety of common environmental noise, which can be mixed on clean speech data. The signal-to-noise (SNR) in dB can be configured as well.

SNR dB = 10 log 10

( P signal P noise
) .

where P signal , P noise are the power of signal and background noise. We set 5 SNR cases here : 40 dB, 30 dB, 20 dB, 10 dB and 0 dB (1 :1 signal vs. noise).

The raw MS-SNSD contains 181 noise files. However, many of them are recorded with strong conversations in other languages (English, German etc.). Some of noise type are also less common. So, these noises are excluded. We'd like to evaluate the effect of noise type on the performance of STT, to make sure that some types are not overpresented, 96 noise files in 18 types are kept.

Evaluation metrics

In Speech-to-Text, the most commonly used metric to evaluate the performance is WORD ERROR RATE (WER).

Other metrics exist, like MATCH ERROR RATE (MER) ; WORD INFORMATION LOST (WIL) or WORD INFORMA-TION PRESERVE (WIP) [START_REF] Morris | From wer and ril to mer and wil : improved evaluation measures for connected speech recognition[END_REF].

W ER = S + D + I N 1 = H + S + D (1) M ER = S + D + I N = H + S + D + I (2) W IP = H N 1 • H N 2 ∼ = I(X, Y ) H(Y ) , ( 3
)
W IL = 1 -W IP (4)
where H, S, D and I correspond to the total number of word hits, substitutions, deletions and insertions. N 1 and N 2 are respectively the number of words in ground-truth text and the output transcripts. The lower are WER, MER and WIL, the better the performance is.

3 Results Speech-to-Text models. Actually, in December 2020 Microsoft introduced background noise suppression functionality in Microsoft Teams meetings [START_REF]Reduce background noise in microsoft teams meetings with ai-based noise suppression[END_REF]. To achieve this, they used 760 hours of clean speech data and 180 hours of noise data. These data are now released for Interspeech 2021 Deep Noise Suppression Challenge [START_REF] Reddy | Interspeech 2021 deep noise suppression challenge[END_REF].

Clean speech

The performance of STT depends also on the noise types.

Main STT errors

The main source of errors contributed to WER is the substitution. For clean speech, or less noisy speech, the percentage of substitution S is generally much higher than deletion D and insertion I. When speech becomes highly noisy (SNR lower than 10 dB), deletion D percentage increased much more. STT service from Microsoft Azure is quite robust to noisy environment, there's practically no change for SNR from 40 dB to 10 dB. Only in the tested case when mixing directly noise and speech 0 dB, the deletion D and substitution S increased slightly. However, the changes are much more significant for other three STT services, especially for IBM Waston. There's also inter-speakers difference of WER. Amount the 42 speakers, all the four STTs had more difficulty to transcribe speech from speaker L23_P08.

Transcription job time

In a production application, the STT service must be as responsive as possible. Google Cloud is the fastest about the four tested APIs, with a median value at Another observation is the server responsiveness : job completion time with Microsoft Azure is almost linear to the audio duration. The variation is also very tight. But for other APIs, though the relationship could be regarded as linear, the variation is much larger. Speeches with same length would require 2 to 4 times more execution time to complete the task.

Discussion

In this work, we evaluated the four most used Speech-to-Text API on French speech from four Computing Cloud : Amazon Transcribe, Google Cloud, IBM Waston and Microsoft Azure. 5 levels of different environment noises are mixed with 6690 clean speeches (17 hours). 100 hours speech tests per STT API gave 400 hours speech transcription.

The results showed that Microsoft Azure's STT service provided the lowest Word Error Rate (median 9%). It's also very robust to common environment noise, even in strong noise environment, the median WERs are only around 16.67%. STT from Amazon Transcribe and Google Cloud performed well, their WER are respectively at 11.76% and 14%. Amazon Transcribe works better in relatively quiet environment while Google Cloud is better for noisy speech. IBM Waston's STT service can provide reasonable results with a median WER at 14.29%. But when the speech is recorded in noisy environment, the WER can go up to around 70% which is difficult to be used. In general, when the signal-to-noise ratio is higher than 20 dB, the WERs are still acceptable. However, if SNR drops lower than 20 dB, except Microsoft Azure, all the three APIs will have difficulties to recognize correctly the speech. Among the 18 environment noise types, Restaurant type is the most difficult one to deal with for all the four STT APIs. When the work is time-constraint, Google Cloud will be the first choice with fastest response time and a reasonable word error rate. Amazon Transcribe can be used when the framework of the project is on the platform of Amazon Web Services. The parallel job can help to reduce the total transcription time, however, per job time is too longer than any other STT service. In average, one transcription job on Amazon Transcribe is 15 times longer than the same job on Google Cloud. Otherwise, the general suggestion will be Microsoft Azure, lowest WER and high robustness to noise. It's more suitable for precision-constraint applications.

FIGURE 1 -

 1 FIGURE 1 -Main topic of WCE-SLT-LIG corpus

FIGURE 2 -

 2 FIGURE 2 -Distribution of WCE-SLT-LIG corpus

FIGURE 3 -

 3 FIGURE 3 -Evaluation on mixed noisy speech by five signal-noise-ratio levels ; upper, wer distributions ; lower, median value for each level. (STTS ACCESSED IN FEBRUARY 2021)

FIGURE 7 -

 7 FIGURE 7 -Transcriptio job time in second for all the four STT systems

TABLE 1 -

 1

Types of background noise used in this work. 96 noises in 18 types

TABLE 2 -

 2 Evaluation on clean audio. Upper, WER distributions ; lower, median values. (STTS ACCESSED IN FEBRUARY 2021)

	50
	40
	30
	WER
	20
			14.29%	14.81%
		11.76%
	10				9.09%
	0
		Amazon Google	IBM	Microsoft
		Amazon Google	IBM	Microsoft
	WER 11.76% 14.29% 14.81%	9.09%
	MER 11.54% 14.29% 14.29%	9.09%
	WIL	0.19	0.25	0.24	0.16

For clean speech, Microsoft Azure performed quite well, with a WER at 9.09% which is close to the advertised rate. Amazon Transcribe took the second place with WER 11.76%. Google Cloud and IBM Waston gave similar WER (14.29% and 14.81%). These WER are actually very good already. According to the public DeepSpeech model

[START_REF] Hannun | Deep speech : Scaling up end-to-end speech recognition[END_REF]

from Mozila, trained with a mixed French dataset "Com-monVoice + CssTen + LinguaLibre + Mailabs + Tatoeba + Voxforge", the WER on test dataset is 19.5% (result retrieved on March 10 th 2021)

[START_REF] Jaco-Assistant | Deepspeech-polyglot[END_REF]

. The gain with cloud STT API is between 24% -53%.

3.2 Noisy speech

After mixing five different levels of environmental noise, Microsoft Azure gave a quite good global WER 11.11% (Tableau 3). Amazon Transcribe and Google Cloud showed the same WER at 20%. But IBM Waston failed at certain point. Its global WER is 29.63%, with a word-informationlost rate at 43% (0.43) which is unfortunately high. The exceptional STT performance of Microsoft Azure is due to that Microsoft has been working intensively on Artificial Intelligence based noise suppression. This environmental noise dataset MS-SNSD comes from Microsoft. The noise suppression should be already in the pipeline of their

  cribe for steaming usage, it's not convenient for non-realtime scenario. One of the potential reasons of the additional seconds from Google Cloud and IBM Waston, could be that Microsoft Azure's returns less complete transcription information than the other three.

																				Amazon
	Percentage %	10 20 30					% of deletion % of insertion % of substitution
		0	clean audio SNR 40dB	SNR 30dB	SNR 20dB		SNR 10dB		SNR 0dB
																				Google
	Percentage %	10 20 30					% of deletion % of insertion % of substitution
		0	clean audio SNR 40dB	SNR 30dB	SNR 20dB		SNR 10dB		SNR 0dB
																					IBM
	Percentage %	10 20 30					% of deletion % of insertion % of substitution
		0	clean audio SNR 40dB	SNR 30dB	SNR 20dB		SNR 10dB		SNR 0dB
																				Microsoft
	Percentage %	10 20 30					% of deletion % of insertion % of substitution
		0	clean audio SNR 40dB	SNR 30dB	SNR 20dB		SNR 10dB		SNR 0dB
	FIGURE 5 -Main transcription errors distribution (mean
	values. The median percentage values for lower SNR are
	zero, less meaningful for presentation)
																	WER by Speakers
		20																																	20 51
	median WER (%)	10 15	15 09	14 29	16 67	12 5	16 29	11 7	15 79	18 18	14 29	15 95	11 54	17 9 16 67	12 82	14 29	13 64	10	12 12	11 11	11 11	11 76 10	10	11 11	13 04	12 5 11 54	11 11	10 53	11 11	10	10 53 9 09	10 62	12 5	10 53	9 72	11 63	11 54 10 3 9 09	1.76 second par
							5					10			15				20			25			30			35			40	job. Microsoft Azure is also fast, 3.51 second per transcrip-
																				Speaker														tion job. IBM Waston is slower and require 5.43 second
																																						to complete the job. It's not surprising that Amazon Trans-
	FIGURE 6 -Word Error Rate by speakers (median values)	cribe is the slowest STT service, with 27 second per job.
	on clean speech for all the four STT systems.							Some transcription jobs can take up to 200 second. Even it's
																																						possible to send up to 100 jobs in parallel, single job wai-
																																						ting is not acceptable for any real world application. This
																																						time requirement does not include the data transfer time to
																																						Amazon S3 storage : with upload speed 100-700 kbps, for a

large amount of data, this can take already quite some time to complete. Though it's possible to call Amazon Trans-