Md Sahidullah Shakeel A Sheikh

Fabrice Hirsch

Slim Ouni

Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning

Keywords: Stuttering detection, speech disorder, data augmentation, class balanced learning

Stuttering is a neuro-developmental speech impairment characterized by uncontrolled utterances (interjections) and core behaviors (blocks, repetitions, and prolongations), and is caused by the failure of speech sensorimotors. Due to its complex nature, stuttering detection (SD) is a difficult task. If detected at an early stage, it could facilitate speech therapists to observe and rectify the speech patterns of persons who stutter (PWS). The stuttered speech of PWS is usually available in limited amounts and is highly imbalanced. To this end, we address the class imbalance problem in the SD domain via a multibranching (MB) scheme and by weighting the contribution of classes in the overall loss function, resulting in a huge improvement in stuttering classes on the SEP-28k dataset over the baseline (StutterNet). To tackle data scarcity, we investigate the effectiveness of data augmentation on top of a multi-branched training scheme. The augmented training outperforms the MB StutterNet (clean) by a relative margin of 4.18% in macro F1-score (F 1 ). In addition, we propose a multi-contextual (MC) StutterNet, which exploits different contexts of the stuttered speech, resulting in an overall improvement of 4.48% in F 1 over the single context based MB StutterNet. Finally, we have shown that applying data augmentation in the cross-corpora scenario can improve the overall SD performance by a relative margin of 13.23% in F 1 over the clean training.

I. INTRODUCTION

S PEECH impairments, often known as speech disorders, are difficulty in producing speech sounds. These speech difficulties usually take the form of dysarthria, cluttering (poorly intelligible speech), lisping, apraxia, and stuttering [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]- [START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF]. Only a few percentage (5-10%) of the world population can produce accurate speech units, the rest encounter some type of speech disorder in their life span [START_REF] Hurjui | Spatial distribution and the prevalence of speech disorders in the provinces of Iran[END_REF]. Of these speech impairments, stuttering is the most predominant one [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Stuttering 1 , is a neuro-developmental speech disorder, in which the flow of speech is disturbed by abnormally persistent and involuntarily speech sounds, which usually take Shakeel A. Sheikh, Md Sahidullah, and Slim Ouni are from Universit é de Lorraine, CNRS, Inria, LORIA, F-54000, Nancy, France (e-mail: shakeel-ahmad.sheikh, md.sahidullah, slim.ouni@loria.fr).

Fabrice Hirsch is from Universit é Paul-Val éry Montpellier, CNRS, Praxiling, Montpellier, France (e-mail: fabrice.hirsch@univ-montp3.fr). 1 Stuttering is also called stammering. In this paper, we will use the terms disfluency, stuttering, and stammering interchangeably.

the shape of core behaviors: including blocks, prolongations and repetitions [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. Stuttering is complex and the several factors that lead to it are delayed childhood development, stress, and speech motor abnormalities [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. In [START_REF] Smith | How stuttering develops: The multifactorial dynamic pathways theory[END_REF], Smith and Weber put forward the multifactorial dynamic pathway theory, where they argued that stuttering occurs due to the failure of the nervous system. People with stuttering (PWS) exhibit impairment in sensorimotor processes which are responsible for the production of speech, and its direction is influenced by emotional and linguistic aspects.

In conventional stuttering detection (SD) and therapy sessions, the speech therapists or speech-language pathologists manually analyze the PWS' speech [START_REF] Nöth | Automatic stuttering recognition using hidden Markov models[END_REF]. The speech therapists observe and monitor the speech patterns of PWS to rectify them [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. This convention of SD is very laborious and timeconsuming and is also inclined toward the idiosyncratic belief of speech therapists. In addition, the automatic speech recognition systems (ASR) are working fine for normal fluent speech, however, they are unsuccessful in recognizing the stuttered speech [START_REF] Mitra | Analysis and tuning of a voice assistant system for dysfluent speech[END_REF], which makes it impractical for PWS to easily access virtual assistants like Apple Siri, Alexa, etc. As a result, interactive automatic SD systems that provide an impartial objective, and consistent evaluation of stuttering speech are strongly encouraged. The SD can also be used to adapt and improve ASR virtual assistant tools for stuttered speech. Despite the fact that having numerous potential applications, very little research attention has been given to the domain of SD and measurement. The detection and identification of stuttering events can be quite a difficult and complex problem due to several variable factors including language, gender, age, accent, speech rate, etc.

The main goal of this work is to build robust automatic stuttering detection systems capable of detecting multiple stuttering types based on speech modality, which later on can be deployed as a tool in real-world settings by providing a means to both PWS and speech therapists to keep track of the stutter speech. These systems can later on be further improved by providing a feedback mechanism to PWS and help them to rectify their stuttering.

A significant amount of work is done in the detection of other speech disorders [START_REF] Verde | Voice disorder identification by using machine learning techniques[END_REF] like dysarthria [START_REF] Narendra | Dysarthric speech classification from coded telephone speech using glottal features[END_REF] and Parkinsons [START_REF] Quan | A deep learning based method for parkinson's disease detection using dynamic features of speech[END_REF], but stuttering has not been addressed widely even though it is the most common one. In this paper, we propose a deep learning framework for robust SD. The automatic detection of stuttering can help in the treatment of stuttering if detected at an early age [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF].

Most of the computer-based SD methods are based either on ASR systems [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF], [START_REF]Sequence labeling to detect stuttering events in read speech[END_REF] or language models [START_REF] Zayats | Disfluency detection using a bidirectional LSTM[END_REF], [START_REF] Chen | Controllable time-delay transformer for real-time punctuation prediction and disfluency detection[END_REF]. These methods are two-stage approaches that first convert the acoustic speech signals into their corresponding spoken textual modality, and then detect stuttering by the application of language models. Even though this ASR-based two-stage approach for identifying stuttering has shown promising results, the dependence on the ASR unit makes it computationally costly and prone to error. Moreover, the adaption towards the ASR task results in the possible loss of stuttering relevant information such as prosodic and emotional content.

In recent decades, the applications of deep learning have grown tremendously in speech recognition [START_REF] Nassif | Speech recognition using deep neural networks: A systematic review[END_REF], speaker recognition [START_REF] Latif | Deep representation learning in speech processing: Challenges, recent advances, and future trends[END_REF], speech synthesis [START_REF] Ning | A review of deep learning based speech synthesis[END_REF], emotion detection [START_REF] Akc | Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers[END_REF], voice conversion [START_REF] Sisman | An overview of voice conversion and its challenges: From statistical modeling to deep learning[END_REF], voice disorder detection [START_REF] Verde | Voice disorder identification by using machine learning techniques[END_REF] including Parkinson's disease detection [START_REF] Almeida | Detecting Parkinson's disease with sustained phonation and speech signals using machine learning techniques[END_REF] and dysarthric speech detection [START_REF] Narendra | Dysarthric speech classification from coded telephone speech using glottal features[END_REF], [START_REF] Kodrasi | Temporal envelope and fine structure cues for dysarthric speech detection using CNNs[END_REF]. Inspired by the human auditory temporal mechanism, Kodrasi et al. [START_REF] Kodrasi | Temporal envelope and fine structure cues for dysarthric speech detection using CNNs[END_REF] recently proposed a convolutional neural network based dysarthric speech detection method by factoring the speech signal into two discriminative representations including temporal envelope (stress, voicing, and phonetic information) and fine structure (breathiness, pitch, and vowel quality) and reported the state-of-the-art results in dysarthria detection. However, the application of deep learning in SD is limited. The acoustic properties of speech disfluencies are different for different disfluencies which can help to discriminate from fluent voice. Due to the presence of these acoustic cues in the stuttered embedded speech, deep learning models can be used to exploit these acoustic cues in the detection and identification of stuttering events. Most of the SD existing methods employ spectral features including mel-frequency cepstral coefficients (MFCCs) and spectrograms or their variants that capture the stuttering-related information.

The earlier studies in this domain applied shallow deep learning approaches in SD. In 1995, Howell et al. [START_REF] Howell | Automatic recognition of repetitions and prolongations in stuttered speech[END_REF] employed two fully connected artificial neural networks for the identification of two types of disfluencies, namely, repetition and prolongation. They extracted autocorrelation features, envelope parameters, and spectral information and used these features as an input to the artificial neural networks for SD. The network was trained on 12 speakers with 20 autocorrelation features and 19 vocoder coefficients. In 2009, Ravikumar et al [START_REF] Ravikumar | Automatic detection of syllable repetition in read speech for objective assessment of stuttered disfluencies[END_REF] proposed multi-layered perceptron for the repetition type of stuttering. The network was trained by using MFCC input features on 12 different disfluent speakers. In 2019, B. Villegas et al. [START_REF] Villegas | A novel stuttering disfluency classification system based on respiratory biosignals[END_REF] trained multi-layer perceptron on 10dimensional respiratory features for the block SD. They used a dataset of 68 Latin American Spanish speakers in their case study. In a recent study, Kourkounakis et al. [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long shortterm memory[END_REF] proposed residual network and bi-directional long short-term memory (ResNet+BiLSTM) based binary deep learning classifiers for the detection of six different types of disfluencies including prolongation, word repetition, sound repetition, phrase repetition, and false starts. They used spectrograms as input features and reported promising results on a small subset (25 speakers) from the UCLASS dataset [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long shortterm memory[END_REF]. In another study Lee et al. [START_REF] Lea | SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] curated a large stuttering dataset (SEP-28k) and utilized the convolutional long short-term memory (ConvLSTM) model for the detection and identification of six types of stuttering, namely, blocks, prolongations, sound repetitions, word repetitions, and interjections. The model takes 40-dimensional input MFCCs, eight-dimensional articulatory features, 41dimensional phoneme feature vector, and three-dimensional pitch features. Sheikh et al. [START_REF] Sheikh | StutterNet: Stuttering detection using time delay neural network[END_REF] recently proposed a single multi-class time delay neural network (TDNN) based Stutter-Net classifier which is capable of detecting core behaviors and fluent speech segments and gives promising detection performance on a larg subset of UCLASS (100+) speakers compared to the state-of-the-art classifiers. The model solely takes 20-dimensional MFCC features as an input. In another recent study M. Jouaiti et al. [START_REF] Jouaiti | Dysfluency classification in stuttered speech using deep learning for real-time applications[END_REF] introduced phoneme-based bi-drectional long short-term memory (BiLSTM) model for SD by mixing the SEP-28k and UCLASS datasets. The model is trained on 20-dimensional MFCCs and 19-dimensional phoneme input features. The disfluencies considered in this work are prolongations, repetitions, and interjections. A detailed summary and comparison of various feature extraction methods and classifiers can be found in [START_REF] Sheikh | Machine learning for stuttering identification: Review, challenges and future directions[END_REF].

This work provides a complete deep learning framework for robust SD following our preliminary initial investigations [START_REF] Sheikh | StutterNet: Stuttering detection using time delay neural network[END_REF] in which StutterNet yields state-of-the-art SD results. In this study, we identify the limitations of StutterNet and proposed further advancements to address those drawbacks. Our main contributions are summarized below:

• Solution for class imbalance problem: The standard stuttering datasets suffer from class imbalance problems. We introduced two strategies based on weighted loss and multi-branch architecture to tackle the class imbalance problem.

• Introducing data augmentation: The stuttering datasets have a limited amount of training data and this makes it difficult to apply advanced deep learning models with a large number of parameters. To address this limitation, this work introduces audio data augmentation. To the best of our knowledge, this is the first work to apply audio data augmentation in the SD problem. • Introducing multi-contextual architecture: Stuttering detection is a special type of speech characterization problem where each class (i.e., stuttering types) has a varying duration. For example, block lasts for a shorter duration than prolongations and repetitions. Therefore, a fixed length of context in the basic StutterNet framework might not be the optimized one for detecting all types of stuttering. We introduce a multi-contextual architecture that uses different context lengths in a parallel fashion. The remaining of the paper is organized as follows. Section II describes the StutterNet and analyzes its deficiencies. Section III discusses the proposed methodology for addressing the deficiencies. Section IV details the experimental design, metrics used and datasets. Section V discusses the experimental results on class balanced, data augmentation, MC StutterNet and cross corpora scenario. Finally, in Section VI, we conclude with the possible future directions.

II. STUTTERNET: OVERVIEW & LIMITATIONS

A. StutterNet

Most of the earlier work employed only a small set of disfluent speakers in their experimental studies, and has approached the SD problem as a binary classification problem: disfluent vs fluent identification or one vs other type [START_REF] Sheikh | Machine learning for stuttering identification: Review, challenges and future directions[END_REF]. The StutterNet we propose in our earlier work, is a time delay neural network based architecture that has been used to tackle the SD as a multi-class classification problem. The StutterNet takes 20 MFCCs as input features with a frame length of 20 ms, mean-normalized with a hop length of 10 ms. Usually, the initial layers in a standard deep neural network learn wider contexts when processing a temporal input signal. However in StutterNet network as shown in Table I, the initial layers learn and capture only smaller contexts and the deeper ones compute the activations from a wider context, thus the deeper/higher layers are able to capture and learn the longer temporal contexts. The network consists of five time delay layers with the first three focusing on [t -2, t + 2], {t -2 , t, t + 2}, {t -3 , t, t + 3} and the other two on {t} contextual frames, respectively. The TDNN layers are having a dilation of (1, 2, 3, 1, 1) respectively. This is followed by a two-layered BiLSTM unit, mean and standard deviation pooling layer, three fully connected (FC) layers, and a softmax layer on top of the network that reveals the classification scores of stuttering disfluencies. A ReLU nonlinearity activation function and a 1D batch normalization are applied after each layer except the statistical pooling layer. We apply dropout of 0.3. in FC layers.

Consider an input speech sample with T frames. The first five layers of StutterNet focus on the small context of speech frames. For example, layer 2 takes as input the sliced output of layer 1 at time frames of {t -2, t, t + 2}, which results in capturing a total temporal context of 9 with the help of the previous layer's context of [t-2, t+2]. Similarly, layer 3 sees a total context of 15 time frames. The statistical pooling layer computes and concatenates the mean and standard deviation by aggregating all T output frames at the end of BiLSTM output. The statistical pooling layer accumulates the information across the temporal dimension which makes it suitable for subsequent layers to operate on the entire temporal speech segment. The StutterNet is trained to classify 5 different types of stuttering including core behaviors and the fluent part of the PWS. For detailed StutterNet architecture, please refer to [START_REF] Sheikh | StutterNet: Stuttering detection using time delay neural network[END_REF].

B. Limitations

Although this network has shown promising results on a SEP-28K dataset in SD, it has several deficiencies. First, it does not generalize quite well on unseen data and leads to overfitting due to the limited amount of data available for training. In addition to data scarcity, the SEP-28k dataset was collected from podcasts in a clean environment, which makes the trained StutterNet difficult to generalize in other environmental conditions. Second, obtaining class-balanced datasets is extremely difficult and very expensive in the speech domain and the SEP-28k dataset is no exception. Deep neural network (DNN) classifiers trained on highly class-imbalanced datasets are usually biased towards the majority class, which

III. ADDRESSING DEFICIENCIES

In this section, we address the three above-mentioned issues.

A. Class Imbalance

In class imbalance learning, the distribution of samples across each class is not uniform. The DNN classifiers, trained on highly imbalanced datasets generally perform poorly for the minority class and favor the majority class [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF]. In the worst case, where there is an extreme imbalance in the training set, the majority class dominates the learning, and samples among the minority class may go undetected, thus affecting performance [START_REF] Weiss | Foundations of Imbalanced Learning[END_REF].

The class-imbalance is one of the major problems in realworld applications, including multi-lingual speech recognition [START_REF] Winata | Adapt-and-adjust: Overcoming the long-tail problem of multilingual speech recognition[END_REF], and stuttering is no different as shown in Fig. 1. In fact, stuttering is extremely imbalanced across fluent and other speech disfluencies. The Fig. 1. shows that the interjections are the most common disfluency present in the SEP-28k dataset followed by repetitions, blocks, and prolongations and the overall distribution is approximately equivalent to the fluent distribution. Collecting a balanced dataset is difficult and expensive for the stuttering detection task. Other datasets such as Kassel State of Fluency (KSoF) (not publicly accessible) [START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: vocalisations, stuttering, activity, & mosquitos[END_REF], FluencyBank [START_REF] Lea | SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] also suffer from this issue.

Over the years, the class imbalance problem is one of the main concerns due to its prevalence, especially in the biomedical domain. Several methods have been proposed to tackle the class imbalance problem, which are mainly categorized into three groups: data-level, cost-level, and architecture-level [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF]. Data-level approaches attempt to re-balance the class distribution by means of some re-sampling methods which include: under-sampling, over-sampling, or combined over-sampling and under-sampling [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF], [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF]. The architecture-level approaches attempt to modify the existing algorithm or develop a new one to tune and adapt it for imbalanced datasets [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF], [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF]. Cost-level approaches attempt to influence the loss/objective function by providing comparatively higher misclassification cost penalties to the minority class in order to force the model to learn about minority classes [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF], [START_REF] Krawczyk | Learning from imbalanced data: open challenges and future directions[END_REF]. In DNNs, addressing the class-imbalance problem by re-sampling may either get rid of sensitive speech samples that are extremely important in training when under-sampling. It can also add numerous quantities of duplicated speech samples under the over-sampling strategy, which eventually makes the training expensive and makes the DNN model likely to overfit [START_REF] Fernández | Learning from Imbalanced Data Sets[END_REF]. Because of these limitations, we investigate cost-level and architecture-level approaches in this work.

For the cost-based approach, we modify the standard cross entropy loss by assigning weights to different classes [START_REF] Cui | Class-balanced loss based on effective number of samples[END_REF]. We set the class weights inversely proportional to the number of samples. We define the weight for class i as

w i = N C * Ni
where N is the number of training samples, C is the number of classes, N i is the number of training samples for class i. Therefore, the weighted cross-entropy (WCE) over the train set can be defined as,

L WCE = 1 B B b=1 M i w i * log(p i ) M i, i∈B w i ( 1
)
where B is the number of batches, M is number of stuttered speech samples in a batch b i ,

p i = e c i C j=1 e c j
is the predicted probability of class c i of sample i.

For architecture-level, we propose a multi-branched approach similar to the work by C. Lea et al. [START_REF] Lea | SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] and M. Bader-El-Den et al. [START_REF] Bader-El-Den | Hierarchical classification for dealing with the class imbalance problem[END_REF] to address the class imbalance issue in SD task. Inspired by the fact that the number of fluent class samples is almost equal to the total number of samples in disfluent classes, we simultaneously classify fluent vs difluent classes in one output branch and subcategories of disfluent classes in another output branch. The multi-branched architecture with two output branches is shown in Fig. 3 (The figure is overall architecture of mulit-contextual StutterNet with two contexts, For single contextual MB StutterNet, only context = 5 is taken into consideration). This has one common encoder E (θ e ) section followed by two parallel branches referred as FluentBranch F (θ f ) and DisfluentBranch D (θ d ). The embeddings from the encoder are processed with both the branches parallelly, where the F is trained to distinguish between fluent and disfluent samples, and the D is trained to differentiate within the disfluent sub-categories.

The objective is to optimize the sum of FluentBranch loss L f and DisfluentBranch loss L d . For simplicity, a simple sum of the two losses has been taken into consideration and it works well. Thus, the overall objective function is define as:

L(θ e , θ f , θ d ) = L f (θ e , θ f ) + L d (θ e , θ d ).
(

) 2
During the evaluation step, if the FluentBranch predicts the sample as fluent, then FluentBranch predictions are considered otherwise DisfluentBranch predictions are taken into consideration to reveal the stuttering category.

B. Data Augmentation

Deep learning has achieved rapid progress in the domain of speech processing tasks including speech recognition [START_REF] Nassif | Speech recognition using deep neural networks: A systematic review[END_REF], speaker recognition [START_REF] Latif | Deep representation learning in speech processing: Challenges, recent advances, and future trends[END_REF], emotion recognition [START_REF] Akc | Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers[END_REF], speech disorder detection [START_REF] Sheikh | StutterNet: Stuttering detection using time delay neural network[END_REF]. However, as a drawback, deep learning based models are hungry for the data and require a substantial amount of annotated data for training, and StutterNet is no exception. The model may require more stuttering data than other speech processing domains, as the presence of disfluencies in stuttering speech corpus is not frequent.

Data augmentation is a popular technique that increases the quantity and diversity of the existing annotated training data, improves robustness, and avoids the overfitting of DNNs. For normal speech recognition, data augmentation demonstrates to be an effective approach for dealing with data scarcity and enhancing the performance of various DNN acoustic methods [START_REF] Park | SpecAugment: A simple data augmentation method for automatic speech recognition[END_REF]. Several data augmentation techniques have been investigated including pitch adjust [START_REF] Meng | Mixspeech: Data augmentation for low-resource automatic speech recognition[END_REF], spectral distortion [START_REF] Kanda | Elastic spectral distortion for low resource speech recognition with deep neural networks[END_REF], tempo perturbation [START_REF] Ko | Audio augmentation for speech recognition[END_REF], speed perturbation, cross-domain adaptation [START_REF] Bell | Transcription of multi-genre media archives using outof-domain data[END_REF], adding noise to clean speech [START_REF] Ko | Audio augmentation for speech recognition[END_REF], spectrogram deformation with frequency and time masking [START_REF] Park | SpecAugment: A simple data augmentation method for automatic speech recognition[END_REF], mixspeech [START_REF] Meng | Mixspeech: Data augmentation for low-resource automatic speech recognition[END_REF], etc.

On the contrary, so far, very limited attention has been given to data augmentation targeting the speech disorder domain. In [START_REF] Vachhani | Data augmentation using healthy speech for dysarthric speech recognition[END_REF], speed and tempo perturbation based data augmentation were used to convert the normal speech to dysarthric speech impairment. In [START_REF] Jiao | Simulating dysarthric speech for training data augmentation in clinical speech applications[END_REF], a voice conversion adversarial training based framework was used to simulate dysarthric speech from healthy speech. In [START_REF] Christensen | Combining in-domain and out-of-domain speech data for automatic recognition of disordered speech[END_REF], normal speech samples (also called out-of-domain) were used as a data augmentation for dysarthric speech in the bottleneck feature extraction stage. In [START_REF] Xiong | Phonetic analysis of dysarthric speech tempo and applications to robust personalised dysarthric speech recognition[END_REF], the speaker-dependent parameter was computed, which was then used for augmentation of scarce dysarthric speech in tempo adjustment. In [START_REF] Woszczyk | Data augmentation for Dementia detection in spoken language[END_REF], several data augmentation techniques such as noise, time stretching, pitch shifting, time shift, masking, etc., were analysed in Dementia detection. There are some studies on data augmentation targeting text based stuttering detection [START_REF] Alharbi | A lightly supervised approach to detect stuttering in children's speech[END_REF], however, in the case of audio based stuttering/disfluency detection, this has not been studied and analysed deeply [START_REF] Grósz | Wav2vec2-based paralinguistic systems to recognise vocalised emotions and stuttering[END_REF] .

In the stuttering domain, the employment of data augmentation is not straightforward, because most of the data augmentations like time stretch, speed perturbation, etc, alter the underlying structure of the stuttering speech sample completely. Our approach employs reverberation and additive noises because it reflects the real-world scenario and does not change significantly the underlying stuttering in the speech sample as shown in Fig. 2. Reverberation consists of convolving speech samples with room impulse responses. We utilize the simulated room impulse responses as described in [START_REF] Ko | A study on data augmentation of reverberant speech for robust speech recognition[END_REF]. For additive noises, we utilize the MUSAN dataset, comprised of 60 hours of speech from 12 languages, 42 hours of music from various genres, and 900 hours of noises [START_REF] Snyder | Musan: A music, speech, and noise corpus[END_REF].

To augment the original speech samples, we combine the training "clean" set with below mentioned augmented copies which results in 5 times increase in training samples.

1) music: A single music sample file randomly chosen from MUSAN is added to the original clean stuttering speech sample (SNR: 5-15dB) (The music sample file is trimmed or repeated as required to match the duration of the clean stuttered speech sample). 2) noise: Throughout the stuttered speech, samples from MUSAN noises are added at 1 sec intervals (SNR: 0-15dB). 3) babble: Speech samples from randomly three to seven speakers are summed together, then added to the original clean stuttered speech sample (SNR: 13-20dB). 4) reverb: The "clean" train set is convolved with simulated room impulse responses. All the data augmentation types shown in Fig. 2 were performed using Kaldi [START_REF] Povey | The Kaldi speech recognition toolkit[END_REF] tool.

C. Multi-contextual StutterNet

The multi-contextual framework is based on the way humans perceive speech. In the cochlea, the input acoustic speech signal is partitioned into several frequency bands so that the information in each band can be filtered independently and thus processed in parallel in the human brain [START_REF] Allen | How do humans process and recognize speech?[END_REF].

Multi-contextual has been studied for action recognition in videos [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF], robust ASR [START_REF] Han | Multistream CNN for robust acoustic modeling[END_REF]- [START_REF] Naderi | Multiresolution convolutional neural network for robust speech recognition[END_REF], speech separation [START_REF] Zhang | A deep ensemble learning method for monaural speech separation[END_REF], where the input speech signal is processed in multiple streams/contexts (multiple time or frequency resolutions), etc. K.J Han et al. Fig. 2: Repetition stuttering with utterance "said that, that" and the effect of various data augmentations (From 0 to 0.25 seconds, the speaker is saying "said", then followed by two repetitions "that" from 0.25 to 0.6 and 1.4 to 1.75).

found that settings like context frame size optimized for one stuttering class are not good for other stuttering types.

Exploiting this fact, we investigate how the multi-contextual neural networks will impact classification performance in the speech disorder domain, and in particular stuttering identification. In our preliminary study, we found that the context window improves the identification performance of two types of disfluencies on the UCLASS dataset [START_REF] Sheikh | StutterNet: Stuttering detection using time delay neural network[END_REF]. As the context frame size increases in the StutterNet, the performance detection of prolongation and repetition also increases, but decreases for fluent speech segments, and almost remains unaffected for a block type of stuttering. The prolongation and repetition last longer than other types of disfluencies. To address this issue, we exploit the variable contexts of MB StutterNet by training the model jointly on different contexts as shown in Fig. 3. The pseudo-code of multi-contextual (MC) StutterNet is provided in Algorithm 1.

Algorithm 1 Pseudo-code for MC StutterNet

L(θe, θ f , θ d ) = L f (θ f ) + L d (θ d ) 15:
Backward Pass: 16:

∇θ b = backward propagation(L(θ), ŷ, d) 17:
Parameter Updates: 18:

θ b ← θ b -λ *

IV. EXPERIMENTAL SETUP

We evaluate our proposed architecture thoroughly on the newly released SEP-28k stuttered dataset [START_REF] Lea | SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], and for cross copora, we use the FluencyBank and LibriStutter datasets.

A. Datasets

SEP-28k: The SEP-28k stuttering dataset was curated from a set of 385 podcasts. The original podcast recordings have varying lengths. From each podcast, 40 to 250 segments (each of length 3 seconds) are extracted which resulted in a total of 28,177 segments. The original SEP-28k dataset contains two types of labels: stuttering and non-stuttering. The stuttering labels include blocks, prolongations, repetitions, interjections, and fluent segments, whereas the nonstuttering labels include unintelligible, unsure, no speech, poor audio quality, and music which are not relevant to our study. In our case study, we use only the stuttering single labeled samples. Out of 28,177 clips, we use only 23573 segments, among which 3286 are repetitions, 1770 are prolongations, 2103 are blocks and 12419 are fluent segments, and 3995 are interjections. This resulted in a total of 19.65 hours of data which includes 2.74 hours of repetition, 1.48 hours of prolongation, 1.75 hours of block, 10.35 hours of fluent speech, and 3.34 hours of interjections. After labeling, each 3-sec sliced speech segment is downsampled to 16 kHz. We randomly select 80% of podcasts (without mixing podcasts) for training, 10% of podcasts for validation, and the remaining 10% of the podcast for evaluation in a 10-fold cross-validation scheme. The speaker information is missing from the SEP-28k dataset, so we divide the dataset based on podcast identities (assuming each podcast is having a unique speaker) 3 .

FluencyBank:

The actual FluencyBank AudioVisual dataset was created by Nan Bernstein Ratner (University of Maryland) and Brian MacWhinney (Carnegie Mellon University) for the study of fluency development. In our case study, we use the annotation done by the Apple similar to SEP-28k [START_REF] Guitar | Stuttering: An Integrated Approach to its Nature and Treatment[END_REF]. This stuttering dataset was curated from 33 podcasts with 23 males and 10 females, which resulted in a total of 4,144 segmented clips. Out of which, we only use 3355 samples (ignoring the non-stuttering and multiple samples), among which 542 are repetitions, 222 are prolongations, 254 are blocks and 1584 are fluent segments, and 753 are Interjections. This results in a total of 2.80 hours of data which includes 0.45 hours of repetition, 0.19 hours of prolongation, 0.21 hours of block, 0.63 hours of interjection samples, and 1.32 hours of fluent samples. We have considered those samples where at least two annotators agree with the same labeling of the segment.

Simulated LibriStutter: The LibriStutter (English) consists of 50 speakers (23 males and 27 females), is approximately 20 hours and includes synthetic stutters for repetitions, prolongations, and interjections [START_REF] Kourkounakis | FluentNet: End-to-end detection of stuttered speech disfluencies with deep learning[END_REF]. Random stuttering was inserted within the four-second window of each speech signal. The original LibriStutter is having 6 classes including fluent, interjection 4 , prolongation, sound, word and phrase repetitions. To make our experiments consistent with the SEP-28k dataset, 3 The details about the train, validation, and test set splits about the SEP-28k dataset is not publicly available. We create a protocol by ensuring no overlap of podcasts between train, validation, and test sets. The details are available in https://shakeel608.github.io/protocol.pdf we treat all repetitions as one class. After extracting samples based on the class label, we did not find any interjection samples in the dataset, so we only train with three classes including fluent, prolongation, and repetitions. For splitting of the dataset, please refer Table 3 in the response sheets.

B. Training Setup

We implement models using the PyTorch library. The acoustic input features used in this case study are 20-dimensional MFCC features, which are generated every 10 ms using a 20 ms window and extracted using the Librosa library [START_REF] Mcfee | Librosa: Audio and music signal analysis in python[END_REF]. For 3-dim pitch and phoneme features, we use Pykaldi [START_REF] Can | PyKaldi: A Python wrapper for Kaldi[END_REF] and Phonexia [START_REF] Fer | Multilingually trained bottleneck features in spoken language recognition[END_REF] tools respectively. For training, we use Adam optimizer and cross-entropy loss function with a learning rate of 10 -2 and batch size of 128. All the results reported in this paper are the average of the 10-fold validation technique and all the training experiments were stopped by early stopping criteria with patience of 7 on validation loss.

C. Evaluation metrics

To evaluate the model performance, we use the following metrics: macro F1-score and accuracy which are the standard and are widely used in the stuttered speech domain [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long shortterm memory[END_REF], [START_REF] Lea | SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF], [START_REF] Sheikh | Machine learning for stuttering identification: Review, challenges and future directions[END_REF], [START_REF] Schuller | The ACM multimedia 2022 computational paralinguistics challenge: vocalisations, stuttering, activity, & mosquitos[END_REF], [START_REF] Sheikh | End-to-end and self-supervised learning for ComParE 2022 stuttering sub-challenge[END_REF]. The macro F1-score (F 1 ) (which combines the advantages of both precision and recall in a single metric unlike unweighted average recall which only takes recall into account) from equation ( 3) is often used in class imbalance scenarios with the intention to give equal importance to frequent and infrequent classes, and also is more robust towards the error type distribution [START_REF] Opitz | Macro F1 and macro F1 a note[END_REF].

F 1 = 1/C k F 1 k = 1/C k 2.P k R k P k + R k ( 3
)
where C is the number of classes and P k , R k , and F 1 k denotes the precision, recall, and F1-score with respect to class k.

D. Experiments

This sections describes briefly the experiments carried out in this paper.

• We carry out experiments using StutterNet with the two state-of-the art SD baselines including ResNet+BiLSTM and ConvLSTM models in the same settings to have a fair comparison. • We perform experiments using weighted loss and multibranched training schemes for addressing class-imbalance problem in stuttering domain, and also exploit the advantage of both the schemes in freezing parts of network. • We experiment with the data augmented training to evaluate its performance in stuttering detection. • We experiment with MC StutterNet on top of data augmentation, and also analyse its performance in crosscorpora settings.

V. RESULTS

In this section, we discuss the results with class balanced training, data augmentation, and multi-contextual learning. We propose two modifications to the vanilla StutterNet to address the class imbalance issue and one of them involves the loss function on the SEP-28k dataset. We further present the results of cross-corpora experiments on the FluencyBank and LibriStutter datasets. Table II shows the results of stateof-the-art baselines (1st part are baselines) and impact of class balanced training. Table III depicts the results using data augmentation on top of class balanced training. Table IV shows the results using MC StutterNet.

Baselines: In addition to our single branch StutterNet baseline (BL5), we first implement the state-of-the-art Con-vLSTM and ResNet+BiLSTM as our baseline models for comparison purposes in the same settings. Our baseline model StutterNet is performing well in almost all the disfluent classes except blocks as compared to the baseline model used in SEP-28k paper [START_REF] Lea | SEP-28k: A dataset for stuttering event detection from podcasts with people who stutter[END_REF] ConvLSTM (MFCC features) (referred to as BL1), ConvLSTM (phoneme features) (referred to as BL2), and ConvLSTM (pitch+MFCC features) (referred to as BL3).

In addition, we also use one more model i.e, ResNet+BiLSTM classifier (referred to as BL4) from Kourkounakis et al. [START_REF] Kourkounakis | Detecting multiple speech disfluencies using a deep residual network with bidirectional long shortterm memory[END_REF]. Comparing our single branch baseline StutterNet to BL4, the model performs well only in interjection and prolongation classes as shown in Table II. For subsequent comparison, we select BL1 (best among BL1, BL2 and BL3) and BL4.

A. Class Imbalance 1) Weighted cross entropy: This section discusses the impact of applying weighted cross entropy to StutterNet (which we call StutterNet WCE ) in order to improve the detection performance of minority classes including repetitions, blocks, and prolongations. Figure 4. illustrates the class-wise training loss curves for normal and weighted cross-entropy loss functions. We can observe that for the standard cross entropy loss function, the majority class (i.e., fluents) exhibit higher loss values and it dominates the overall loss.

Therefore, during training with backpropagation, the number of updates for fluent class dominates the gradient values that in turn forces the model to focus mainly on correctly classifying/predicting the majority class. Thus, the minority classes including the blocks, prolongations, and repetitions are given less importance during training, which leads to their poor detection performance. Table II confirms that the detection performance of blocks, prolongations, and repetitions is very poor, as they are mostly predicted as fluent samples due to the class imbalance nature of the problem. The loss functions for the weighted cross entropy are shown by dashed curves in Fig. 4. The figure indicates that applying weights to standard cross entropy loss function by equation ( 1) forces the model to give balanced importance to each disfluency class while optimizing the parameters of the network during backpropagation. This, in turn, helps in boosting the gradient updates for minority classes during training, and thus increases their detection performance in baselines including BL4, and BL5 as shown in Table II. The StutterNet WCE gives a relative improvement of 33.49%, 83.50%, 1,185%, and 5.98% over BL5 and 58.69%, 462%, 500%, and 7.84% over BL1 for detecting repetitions, prolongations, blocks, and interjections, respectively. Table II also demonstrates the suitability of WCE with competitive ResNet+BiLSTM method and results in 54%, 57.35%, 517%, and 10.23% relative improvement in repetitions, prolongations, blocks, and prolongations over BL4.

2) Multi-branch training: Moreover, we address the class imbalance problem via a multi-branch network (referred to as StutterNet MB ). This has two output branches: FluentBranch and DisfluentBranch as shown in Fig. 3. (For StutterNet MB , only a single context of five is taken into consideration) This method improves the detection performance of repetitions, prolongations, blocks by a relative margin of 29.92%, 0.65%, 144% respectively over BL5, and 54.45%, 208%, 13.72%, 6.80% in repetitions, prolongations, blocks, and fluents respectively over BL1, and 88%, 31.81% in repetitions, blocks respectively over BL4, however, the BL4 is performing better in prolongations and interjection classes. We also applied multi-branch training in ResNet+BiLSTM which results in a relative improvement of 85.45% in repetition class only as compared to the baseline BL4.

In applying class balanced training, we found that there is drop in macro F 1 score. Using StutterNetWCE and WCE based ResNet+BiLSTM, the macro F 1 score drops from 42.84% and 43.12% to 41.02% in StutterNet and 41.02% in ResNet+BiLSTM respectively. By employing multi-branch training, the macro F 1 score drops very slightly from 42.84% 3) Analysis of confusion matrix: Using the WCE training scheme, the detection performance of minority classes improved remarkably at the cost of fluent accuracy. From the Fig. 5 (a), we analyze that most of the repetitions and blocks are being classified as fluent speech. Initially, we hypothesize that it is most likely because of the imbalanced nature of the problem. Despite addressing the class imbalance problem in the stuttering domain using WCE and multi-branched training, we found that the block and repetition type of stuttering disfluencies are the ones that are still getting confused with the fluent class as depicted in Fig. 5 (b) and Fig. 5 (c). This makes intuitive sense because the blocks are closely related with fluent segments having just different initial silence or gasp followed by fluent utterances like fluent speech. The repetitions, on the other hand, contain some word or phrasal repetitions, which are actually fluent utterances, if we carefully analyze their individual parts. Consider an utterance he he is a boy. The word he is being repeated twice but is a fluent part if two he's can be analyzed on an individual basis.

4) Exploiting advantage of WCE and MB StutterNet: Since StutterNet WCE and StutterNet MB address the class-imbalance issue differently, we combine them to exploit both of their advantages. We first pre-train the StutterNet WCE and use it as a DisfluentBranch in our multi-branched StutterNet. After the pre-training step, we freeze parameters in two ways. First, we freeze the parameters of the contextual encoder only and fine-tune only the two output branches. We label this training scheme as M frz enc . By exploiting this method, we achieve an overall detection improvement of 5.11% in F 1 over StutterNet MB . Second, we freeze the base encoder and StutterNet WCE (DisfluentBranch) and append with one more FluentBranch (to distinguish between fluents and stutter samples). We refer this as M frz enc,disf and it results in an overall improvement of 6.01% in F 1 over StutterNet MB .

We also experiment by first training the model using weighted cross entropy in FluentBranch, and then by finetuning the DisfluentBranch by freezing the parameters of the encoder and FlunetBranch. We refer to it as M frz enc,fluent and the results for this configuration are shown in Table II and III. However, this training scheme degrades performance in almost all the disfluent classes in comparison to M frz enc (freezing encoder only) and M frz enc,disf (freezing encoder and DisfluentBranch). This is possibly due to the reason that the base encoder is trained only to distinguish between fluent and disfluent classes via FluentBranch. Then freezing its parameters in a fine-tuning step further inhibits it more in learning sub-classes (repetitions, prolongations, interjections, and blocks) of the disfluent category which makes their overall detection performance lower.

B. Data augmentation

The main experimental results obtained with various data augmentation techniques are shown in Table III, where we compare the detection performance obtained with data augmented training to the baseline clean dataset. We first separately train the StutterNet MB on different data augmentation techniques which are described in Section III. We found that the training MB ResNet+BiLSTM and StutterNet MB with Kaldi augmentation increases the overall F 1 performance. From Table III improvement of 7.28%, and 6.81% in overall accuracy respectively. In addition to Kaldi augmentation, we also apply pitch scaling and bandpass filter as data augmentation, however, we did not achieve much improvement in SD.

C. Multi-contextual StutterNet

Different contexts show different optimized class accuracies. In order to exploit these different variable contexts and to improve the detection performance further, we propose a multi-contextual (MC) StutterNet for SD as shown in Fig. 3. We jointly train and optimize the MC StutterNet on the clean and augmented data using variable contexts of 5 and 9. The embeddings extracted from each context as depicted by CB block are passed to a two-layered BiLSTM unit, and then concatenated after applying statistical pooling layer (SPL), resulting in a 1×2×(2×N )-dimensional feature vector (where N is layer size), which is then fed parallelly to two different branches including FluentBranch and DisfluentBranch for class predictions. This results in a relative improvement of 13.40%, 15.67%, 6.41%, and 1.38% in prolongation, block, interjection, and fluent classes over StutterNet MB (clean), thus an improvement of 3.79% in macro F 1 score, however, employing multi-contextual training, we see a drop from 35.26% to 33.36% in repetition accuracy over the StutterNet MB . In comparison to baseline BL5 (vanilla single branch StutterNet), there is a relative improvement of 22.92%, 14.13%, 181.81%, 3.27% in repetition, prolongation, block, and interjection classes respectively. Thus a relative improvement of 2.38% in macro F 1 score. The MC StutterNet also performs better

D. Summary of proposed methods

This work advances the basic StutterNet by addressing its limitations with three modifications. In Table V, we present a summary of the results demonstrating systematic improvements. We observe that all the proposed modifications help to gradually improve the performance and we achieve 7.37% overall relative improvement in terms of macro F 1 score.

E. Cross corpora evaluation

Table VI shows the results on cross corpora datasets including FluencyBank and simulated LibriStutter. By optimizing on SEP-28k dataset in terms of data augmentation and multicontextual, we aim to evaluate our proposed methodology MC StutterNet on a cross corpora scenario. We train StutterNet on SEP-28k dataset and evaluate it on FluencyBank dataset which comprises samples from 33 podcasts. We found that the model trained on one corpus (SEP-28k) fails to generalize and performs poorly on cross-domain corpora. As can be seen from the Table VI and Fig. 6, the F 1 detection performance decreases remarkably from 43.86% to 38.92% employing clean training. We hypothesize that the performance drop is due to the domain mismatch due to the difference in speaker accent and recording environment between the SEP-28k and FluencyBank datasets. The repetitions and block classes show more degradation in their performance in cross corpora scenarios. Furthermore, we apply data augmentation in cross corpora evaluation, and we found that it boosts the detection performance of all the classes, which results in an improvement of 13.23% in F 1 .

Experimental evaluation of MC StutterNet on simulated LibriStutter dataset [START_REF] Kourkounakis | FluentNet: End-to-end detection of stuttered speech disfluencies with deep learning[END_REF] results in a macro F 1 score of ≈ 91% (shown in Table VI). However, the performance considerably drops when evaluated in cross-corpora setting in comparison to real stuttering dataset FluencyBank. The MC StutterNet trained on SEP-28k dataset shows extremely poor performance when tested on simulated LibriStutter dataset. Applying data augmented training, we see there is minimal improvement of 1% and 2% in repetition and prolongation, respectively. The LibriStutter simulated dataset does not reflect the actual nature and characteristics of stuttered speech. The results also confirms that a model trained on simulated stuttered datasets should not be used in a real clinical condition.

VI. CONCLUSION

This paper addresses the problem of class imbalance in the stuttering domain. We address the class imbalance problem via two strategies including weighted cross entropy and a multi-branch training scheme. The weighted cross entropy loss function forces the StutterNet classifier to give more attention to minority classes. We also investigate the effectiveness of data augmentation in the SD domain. For data augmentation, we employ reverberations and additive noises from the MUSAN dataset [START_REF] Snyder | Musan: A music, speech, and noise corpus[END_REF]. Additionally, we propose a MC StutterNet, time delay based neural network for SD. The proposed MC StutterNet is a multi-class classifier that exploits different variable contexts trained jointly on different contexts (5, 9) using CE. More importantly, the experiments using data augmentation over the FluencyBank dataset revealed that our methodology generalizes better in the cross corpora domain.

For class imbalance, we have used only a simple weighting scheme in the cross-entropy loss function, which results in an accuracy trade-off between majority and minority classes.

In general, the data augmentation helps in stuttering domain, however, the use of data augmentation in the stuttering domain is not straightforward, thus, is limited because most data augmentations, such as time stretch, speed perturbation, and so on, completely alter the underlying structure of the stuttering speech sample. In order for data augmentation to be more effective, a domain stuttering specific data augmentation is required to be developed. In addition, the stuttering detection domain has not matured enough, so a single metric like other speech domains which reflects the overall performance of a model is yet to be developed. In addition to accuracy metric, we have also used a macro F1-score (F 1 ) which gives a good indication for the better evaluation of proposed methods. Moreover, we use joint training over multi contexts in this work, and it is possible that one context can dominate the training. A visualisation summary of macro F 1 score of models in stuttering detection is shown in Fig. 7.

The proposed methodology show promising results and it can detect if the stuttering is present in the speech sample or not, however, it cannot predict where exactly the stuttering occurs in the speech frames. For future study, we would like to explore combining the other different types of neural networks in SD to predict the frames where exactly the stuttering occurs. In addition to varying context, the investigation of varying depth and different number of convolutional kernels is also an interesting topic to study in SD. Moreover, the temporal information captured by recurrent neural networks can also be investigated in a multi-stream fashion for the identification of disfluent speech frames.

The performance comparison of the proposed systems with two state-of-the-art systems demonstrate that even though we achieve a noticeable advancements, the automated stuttering detection requires further research for developing a clinically usable system. The stuttering detection is fundamentally a challenging task due to inter-person variations, language/accent/dialect variability and other speaking variations. The scopes of the current work are limited to addressing basic problems related to stuttering detection. This work can be extended with speaker-adaptive training, domain adaptation to further improve the stuttering detection problem.

Regarding cognitive aspects of stuttering are well modeled or not as the explainability analysis of the used deep models along with auxiliary data (e.g., functional magnetic resonance imaging or fMRI data) is yet to be explored. However, we think the base architecture (i.e., MC StutterNet) developed in this work could still be useful for such explainability analysis. The stuttering detection can be improved with multimodality by integrating with visual cues related to head nodding, lip tremors, unusual lip shapes, quick eye blinks, facial expres-sions, etc. We think that the same base acoustic model can be used in a fusion framework.

We have found that the blocks are the most difficult to detect. It would be interesting to analyze the disfluencies of the speakers which are hardest to identify by doing more ablation analysis. The future work includes exploring self-supervised models that exploit unlabelled audio data.

Fig. 1 :

 1 Fig. 1: Stuttering data distribution in SEP-28k dataset showing five classes consisting of single stuttering class.

  [START_REF] Han | Multistream CNN for robust acoustic modeling[END_REF] recently proposed a multi-stream 2 convolutional neural network for robust acoustic modeling. Chiba et al.[START_REF] Chiba | Multi-stream attention-based BLSTM with feature segmentation for speech emotion recognition[END_REF] recently proposed multi-stream attention-based BiL-STM network for speech emotion recognition. Li et al.[START_REF] Li | Using multi-stream hierarchical deep neural network to extract deep audio feature for acoustic event detection[END_REF] extracted deep features by training multi-stream hierarchical DNN for acoustic event detection. Moreover Sheikh et al.[START_REF] Sheikh | StutterNet: Stuttering detection using time delay neural network[END_REF]

Output:stat 9 =

 9 Predicted label set ŷ ∈ (R, P, B, In, F ) Input: Stutter dataset with D = (X i , f i , d i ), where each sample X i is R 20×T MFCC input sequence d: Stutter label L: Total loss f : Pseudo label for FluentBranch b: Sample batch K: Number of epochs C: Context CE: Cross entropy loss function λ: Learning rate p f : Prediction of FluentBranch θ = [θe, θ f , θ d ] : Network parameters p d : Prediction of DisfluentBranch θe : Encoder parameters L f : Loss of FluentBranch θ d : DisfluentBranch parameters L d : Loss of DisfluentBranch θ f : FluentBranch parameters Ensure: MC StutterNet weight initialization 1: for i in K do 2: for b in D do 3: StatisticalPooling(out 9 ) 9: stat embed = Concat(stat 5 , stat 9 ) 10: p f = FluentBranch(stat embed ) 11: p d = DisfluentBranch(stat embed ) 12: L f = CE(p f , f b ) // f b : Pseudo label of a batch 13: L d = CE(p d , d b ) // d b : Stutter label of a batch 14:

4 Fig. 3 :

 43 Fig.3: A schematic diagram of Multi-contextual StutterNet, which is a multi-class classifier that exploits different variable contexts of[START_REF] Kehoe | Speech Language Pathology-Stuttering[END_REF][START_REF] Mitra | Analysis and tuning of a voice assistant system for dysfluent speech[END_REF] in SD. The FluentBranch and DisfluentBranch are composed of 3 fully connected layers followed by a softmax layer for prediction of different stuttering classes, CB: Context Block, SPL: Statistical Pooling Layer. The context C (5, 9) here does not mean the TDNN layers, but rather it means the kernel-size which is number of frames taken into account when processing speech frames (C=5 means a context of 5 frames are taken at a time and C =9 means a context of 9 frames are taken at a time). These two contexts are jointly exploited in MC StutterNet as shown in the left hand side of the figure.

Fig. 4 :

 4 Fig. 4: Class-wise loss values for different probabilities of ground truth classes. Here block, fluent, repetition, prolongation, and interjection classes are correspondingly denoted by B, F, R, P, and I. The standard cross entropy (CE) is shown by solid curves and its weighted version is shown by dashed ones (WX represents the weighted loss of class X).

( a )Fig. 5 :

 a5 Fig. 5: Accuracy confusion matrices showing the confusion of fluent speech with repetitions and blocks. F: Fluent, R: Repetition, B: Block, P: Prolongation, In: Interjection, BL: Baseline, WCE: Weighted cross-entropy, MB: Multi-branch.

Fig. 7 :

 7 Fig. 7: Macro F 1 score summary of proposed and baseline models (BL1 and BL4). The red, light blue and purple bars indicate the baseline, same-corpora setting, and cross-corpora settings. (CC: Cross Corpora with training on SEP-28k and evaluated on FluencyBank dataset, A4: All four data augmentation. The second last bar shows cross-corpora performance on FluencyBank when trained using a clean SEP-28k dataset and the last bar shows the same with data augmentation).

TABLE I :

 I STUTTERNET ARCHITECTURE, TC: TOTAL CON-TEXT, TDNN: TIME DELAY NEURAL NETWORK LAYER, FC: FULLY CONNECTED LAYER, BILSTM: BIDIRECTINAL LONG SHORT-TERM MEMORY (2 LAYERS), LAYER CON-TEXT OF [T-2, T+2] MEANS 5 FRAMES ARE TAKEN INTO CONSIDERATION TWO BEFORE THE CURRENT TIME STEP AND TWO AFTER THE CURRENT TIME STAMP.

	Layer	Output Layer Size	Layer Context	TC
	TDNN 1	64	[t -2, t + 2]	5
	TDNN 2	64	{t-2, t, t+2}	9
	TDNN 3	64	{t-3, t, t+3}	15
	TDNN 4	64	{t}	15
	TDNN 5	3*64	{t}	15
	BiLSTM	64 * 2	{t}	T
	Statistical Pooling	3 * 64 * 2×1	[0, T )	T
	FC1	64	-	T
	FC2	64	-	T
	FC3	NumClasses	-	T
	results in poor modeling of minority classes [32]. In addition
	to the above deficiencies, we found in our previous work Stut-
	terNet [29] that the context is very important in SD. Stuttering
	detection is a special type of speech characterization problem
	where each class (i.e., stuttering types) has a varying duration.
	For example, block lasts for a shorter duration than prolonga-
	tions and repetitions. Therefore, a fixed length of context in
	the basic StutterNet framework might not be the optimized
	one for detecting all types of stuttering. A larger context
	increases the performance of prolongation and repetition types
	of disfluencies but decreases the recognition performance of
	fluent speech segments on the UCLASS dataset [29].

TABLE II :

 II RESULTS WITH BASELINES (BL) AND USING CLASS IMBALANCE LEARNING (CLEAN TRAINING) (B: BLOCK, F: FLUENT, R: REPETITION, P: PROLONGATION, IN: INTERJECTION, TA: TOTAL ACCURACY, F 1 : MACRO F1-SCORE), F MFCC : MFCC INPUT FEATURES, F F0 : 3-DIM (PITCH, PITCH-DELTA, VOICING) FEATURES, F phone : PHONEME FEATURES, MB: MULTI BRANCH, WCE: WEIGHTED CROSS ENTROPY, M frz enc :FREEZING ENCODER, M frz enc,disf : FREEZING ENCODER AND DISFLUENTBRANCH, M frz enc,fluent : FREEZING ENCODER AND FLUENTBRANCH.

				Accuracy
	Method	R	P	B	In	F	TA	F 1 (%)
					Baselines
	ConvLSTM + F MFCC (BL1) [28]	22.83 10.61 06.34 56.74 72.35 52.68	34.00
	ConvLSTM + F phone (BL2) [28]	10.18 01.06 00.35 43.88 74.48 48.43	24.00
	ConvLSTM + F F0+MFCC (BL3) [28] 19.28 09.55 08.51 51.78 66.60 48.47	30.80
	ResNet+BiLSTM (BL4) [27]	18.76 41.24	5.47	57.18 88.19 62.36	43.12
	StutterNet (BL5) [29]	27.14 32.55	2.96	57.74 87.60 62.57	42.84
					Class Imbalance
	ResNet+BiLSTM + WCE [27]	28.90 64.89 33.79 63.03 46.90 47.42	41.00
	MB ResNet+BiLSTM [27]	34.79 30.19	5.92	49.26 75.47 55.62	39.20
	StutterNet + WCE (StutterNet WCE )	36.23 59.73 38.05 61.19 41.59 45.26	41.02
	MB StutterNet (StutterNet MB )	35.26 32.76	7.21	56.04 77.27 58.56	42.26
	M frz enc	39.82 37.91 10.45 60.57 73.49 58.58	44.42
	M frz enc,disf	29.25 45.85 18.11 56.88 74.49 58.18	44.80
	M frz enc,fluent	31.15 27.62 05.01 57.64 73.64 55.83	38.60

TABLE III :

 III RESULTS USING DATA AUGMENTATION (B: BLOCK, F: FLUENT, R: REPETITION, P: PROLONGATION, IN: INTERJECTION, TA: TOTAL ACCURACY, F 1 : MACRO F1-SCORE), A4: BABBLE + REVERBERATION + MUSIC + NOISE AUGMENTATION, MB: MULTI BRANCH, WCE: WEIGHTED CROSS ENTROPY, M frz enc :FREEZING ENCODER, M frz enc,disf : FREEZING ENCODER AND DISFLUENTBRANCH, M frz enc,fluent : FREEZING ENCODER AND FLUENTBRANCH.

				Accuracy
	Method	R	P	B	In	F	TA	F 1 (%)
	StutterNet + A4	28.67 32.53	3.69	64.24 88.70 64.69	45.30
	StutterNet + WCE (StutterNet WCE ) + A4 43.06 61.43 38.18 65.33 43.73 48.30	44.34
	MB ResNet+BiLSTM + A4	33.27 32.85	0.99	65.95 70.92 55.75	39.44
	MB StutterNet (StutterNet MB ) + A4	34.05 33.93	4.32	68.74 78.88 61.35	44.03
	M frz enc + A4 M frz enc,disf + A4	28.87 30.94 27.63 36.33 11.52 62.31 83.05 62.14 4.26 64.12 85.29 62.85	44.06 45.76
	M frz enc,fluent + A4	31.19 25.14 04.48 61.30 77.56 58.32	39.80

  , it can be seen that the data augmentation does help in improving the F 1 in almost all the cases. Applying data augmentation with a single branched and StutterNet WCE , there is a relative improvement of 5.74% and 3.50% in F 1 respectively, over the clean versions of the training. When data augmentation is applied to MB training, there is a relative improvement of 4.17% and 0.61% in F 1 using StutterNet MB and ResNet+BiLSTM over clean training, respectively. Moreover, applying Kaldi data augmentation on top of the M frz enc and M frz enc,disf training scheme, there is a relative The bar plot clearly shows that the model MC StutterNet trained on clean SEP-28k dataset fails to generalize on FluencyBank cross corpora data. Applying data augmentation improves the stuttering detection in cross domain corpora as shown orange bars (4 th column in each disfluency).

	MC SC:Clean 1
	80	MC SC:A4 2
	MC CC:Clean 3
		MC CC:A4 4
	60
	40
	20
	0
	RA	PA	BA	InA	FA	F 1
	Fig. 6: Impact of data augmentation (A4) in cross corpora
	FluencyBank dataset with MC StutterNet (R: Repetition, P:
	Prolongation, B: Block, In: Interjection, F: Fluent, XA: X:
	Disfluency Class and A: Accuracy, MC: Multi contextual
	StutterNet, SC: Same corpora, CC: Cross corpora, A4: Aug-
	mentation).

TABLE IV :

 IV RESULTS USING MC StutterNet WITH CLEAN AND DATA AUGMENTATION. THE MC StutterNet ALSO CONTAINS MB TRAINING. (B: BLOCK, F: FLUENT, R: REPETITION, P: PROLONGATION, IN: INTERJECTION, TA: TOTAL ACCURACY, BB: BABBLE, RV:REVERBERATION, MU:MUSIC, NO:NOISE, A4: BB + MU + NO + RV AUGMENTATION)

				Accuracy
	Method	R	P	B	In	F	TA	F 1 (%)
	MC StutterNet (Clean) 33.36 37.15 08.34 59.63 78.34 59.77	43.86
	MC StutterNet + Bb	32.29 37.11 09.85 66.50 74.54 58.71	44.40
	MC StutterNet + Mu	32.08 36.57 07.98 65.68 77.67 59.96	44.00
	MC StutterNet + No	28.54 32.39 04.75	64.20 80.42 59.99	42.80
	MC StutterNet + Rv	31.98 37.02 05.83 67.63 77.41 60.08	44.20
	MC StutterNet + A4	34.65 39.75 10.12 67.91 77.89	61.72	46.00

TABLE V :

 V SUMMARY OF RESULTS OF PROPOSED METHODS (A4: ALL FOUR AUGMENTATION (BB + MU + NO + RV) can be observed in Table IV. By applying all four data augmentation in MC StutterNet training, there is an overall improvement of 1.76%, 17.15%, and 134% in repetitions, prolongations, and blocks respectively as shown in the Table IV, which is a 4.48% relative gain in macro F 1 score over the augmented StutterNet MB training.

	Method	F 1 (%)
	StutterNet (BL5) (Clean) [29]	42.84
	Class Imbalance
	StutterNet WCE (Clean)	41.02
	StutterNet MB (Clean)	42.26
	M frz enc (Clean) M frz enc,disf (Clean)	44.42 44.80
	Data Augmentation
	StutterNet + A4	45.30
	StutterNet WCE + A4	44.34
	StutterNet MB + A4	44.03
	M frz enc + A4 M frz enc,disf + A4	44.06 45.76
	Multi-Contextual
	MC StutterNet (Clean)	43.86
	MC StutterNet + A4	46.00
	in macro F 1 score in comparison to state-of-the-art baselines
	BL1 and BL4. Applying data augmentation on top of the MC
	StutterNet, we found that except noise augmentation, all other
	data augmentation types help in improving the macro F 1 score
	in comparison to MC StutterNet (clean) The noise augmented
	samples help in improving the detection accuracy of fluent
	class. For interjections, we found that all the data augmentation
	helps, and for blocks, only babble augmentation helps in
	improving their detection accuracy. We also found applying
	all four data augmentation techniques in MC StutterNet results
	in an accuracy improvement in prolongation and repetition
	classes, however, with individual data augmentation, a drop
	in their accuracies

TABLE VI :

 VI RESULTS ON CROSS CORPORA FLUENCYBANK AND SIMULATED LIBRISTUTTER DATASETS. THE FIRST ROW IS FROM TABLE IV WHICH SHOWS THE RESULTS ON THE SAME CORPORA SEP-28K DATASET. THE LAST TWO ROWS IN THE FIRST HALF OF THE

  TABLE SHOW THE RESULTS, WHERE THE MODEL IS TRAINED ON SEP-28K AND TESTED ON FLUENCYBANK IN CROSS CORPORA SETTING. (B: BLOCK, F: FLUENT, R: REPETITION, P: PROLONGATION, IN: INTERJECTION), BB: BABBLE, RV: REVERBERATION, MU: MUSIC, NO: NOISE, A4: ALL FOUR BB + RV + MU + NO AUGMENTATION. THE MULTI-BRANCHED TRAINING IS ALSO CONSIDERED IN MC StutterNet.

															Accuracy
				Method			TrainSet		TestSet	R	P	B	In	F	TA	F 1 (%)
			MC StutterNet		SEP-28k		SEP-28k	33.36 37.15 08.34 59.63 78.34 59.77	43.86
			MC StutterNet (Clean)	SEP-28k		FluencyBank 19.48 35.80 01.83 56.36 80.85 56.48	38.92
			MC StutterNet + A4		SEP-28k		FluencyBank 22.54 42.22	4.36	64.56 84.04 60.92	44.07
			MC StutterNet		LibriStutter		LibriStutter	93.36 76.26	NA	NA	98.19 96.11	91.00
			MC StutterNet(clean)		SEP-28k		LibriStutter	00.65 00.48	NA	NA	99.95 77.25	30.00
			MC StutterNet + A4		SEP-28k		LibriStutter	01.11 02.39	NA	NA	97.35 75.43	31.00
			MC StutterNet		LibriStutter		SEP-28k	24.24 55.11	NA	NA	60.60 53.21	41.00
	46
	44
	42
	40
	38
	36
	34
	BL1	BL4	StutterNet WCE	StutterNet WCE + A4	StutterNet MB	StutterNet MB + A4	M frz enc	M frz enc + A4	M frz enc,disf	M frz enc,disf + A4	MC StutterNet	MC StutterNet + A4	MC StutterNet CC	MC StutterNet CC + A4

multi-stream, multi-scale, multi-resolution are the different names of multicontext.

ACKNOWLEDGMENT

This work was made with the support of the French National Research Agency, in the framework of the project ANR BENEPHIDIRE (18-CE36-0008-03). Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several universities as well as other organizations(see https://www.grid5000.fr) and using the EXPLOR centre, hosted by the University of Lorraine.


Caroline Chubilleau

Bertrand Lenoir
email: bertrand.lenoir@ijl.nancy-universite.fr

Philippe Masschelein

Anne Dauscher

Christophe Candolfi

E Guilmeau

C Godart

Influence of ZnO nano-inclusions on the transport properties of the CoSb3 skutterudite

Keywords: thermoelectric materials, composite materials, microstructure, thermoelectric properties

come

Introduction

The search for thermoelectric materials with higher efficiency is currently of prime interest in the field of renewable energy. Such materials can either convert waste heat (coming from sun, geothermal energy, cars or industrial processes…) to useable electricity, or can act as solid-state heat pumps when supplied with electrical current. The effectiveness of a material for thermoelectric applications is governed by the dimensionless figure of merit ZT, given by [START_REF] Rowe | Thermoelectrics Handbook: Macro to Nano[END_REF] with T the absolute temperature and the following transport parameters: the Seebeck coefficient or thermopower S, the electrical resistivity r, the total thermal conductivity where ke is the electronic thermal conductivity and kL is the lattice thermal conductivity [START_REF] Rowe | Thermoelectrics Handbook: Macro to Nano[END_REF]. Since S, r and ke are all sensitive to the carrier concentration, optimizing ZT is an extremely challenging task. Therefore, the ZT values of state-of-the-art thermoelectric materials based on Bi1-xSbx, Bi2Te3, PbTe, and Si1-xGex have been limited to unity for several decades [START_REF] Rowe | Thermoelectrics Handbook: Macro to Nano[END_REF][START_REF] Lenoir | Semiconductors and Semimetals[END_REF][START_REF] Dauscher | [END_REF].

Quite recently, skutterudite compounds have emerged as potential candidates for thermoelectric applications at high temperature [START_REF] Uher | Semiconductors and Semimetals[END_REF]. The general chemical formula of binary skutterudites is MX3, where M is a transition metal such as Co, Rh or Ir, and X stands for a pnictogen i.e. As, P or Sb. Some of them, such as CoSb3, show good electrical properties (high carrier mobility and large effective mass) comparable to those of Bi2Te3 and PbTe.

However, the large lattice contribution to the total thermal conductivity (~ 10 W.m -1 .K -1 at 300 K for CoSb3) is the main obstacle preventing to achieve high ZT values [START_REF] Morelli | [END_REF]. The presence of voids in the crystal structure of these compounds opened up a novel way to lower the € ZT = S 2 T ρ (κ e + κ L ) κ = κ e +κ L phononic part. Slack [START_REF] Slack | CRC Handbook of Thermoelectrics[END_REF] suggested to fill these voids with weakly bound atoms that "rattle" about their equilibrium position. These voids can accommodate a large number of elements resulting in ternary derivatives AxM4X12 referred as 'filled skutterudites' [START_REF] Braun | [END_REF]. The presence of a filler atom in these cages induces an additional phonon scattering mechanism, usually modeled by an Einstein-like mode, which strongly dampens the lattice thermal conductivity [START_REF] Morelli | [END_REF]8]. This approach proved to be an efficient route to push the ZT values beyond unity [START_REF] Uher | Semiconductors and Semimetals[END_REF][8][9][10][11].

Introducing nano-scaled structures into the bulk phase matrix is another approach for improving the thermoelectric properties that has received attention over the last years.

Nanostructured materials have theoretically the potential to both enhance the electrical properties and limit the thermal conduction. The former feature is related to quantum confinement effects resulting in higher thermopower values [12]. The second property arises from the limitation of the phonon mean free path due to the presence of numerous nanoscopic interfaces. Several strategies have been developed to produce nanostructured CoSb3 either by grain size reduction [13] or by dispersion of nanoparticles in the micrometric grained-sized matrix [14][15][16] or by creating in situ stable nanoinclusions [17]. Most of these last studies used mechanical mixing to introduce the nanoinclusions. These studies also mainly focused on filled skutterudites for which both the filler atoms and the nanoparticles modify the transport properties. The influence of the former thus obscures the role played by the latter preventing the underlying physical mechanisms to be investigated in detail. Such an approach was the starting point of the study of Su et al. [18].

Here, we report on the addition of ZnO nanoinclusions in the binary skutterudite CoSb3 with the aim of determining the exact role played by stable oxide nanoparticles on the thermoelectric properties. The ZnO-CoSb3 powders, fabricated using an ultrasonic mixing in liquid combined to freeze-drying, were densified by spark plasma sintering (SPS). Five samples of CoSb3 + x wt% ZnO were synthesized with x = 0, 2, 4, 8 and 12. The transport properties (including electrical, thermal and galvanomagnetic properties) of these specimens were investigated in a broad temperature range (2 -800 K) and are discussed in light of the chemical and microstructural characterizations.

Experimental procedures

Materials synthesis

The preparation of the CoSb3 + x wt% ZnO composites used in this study requires several steps. The first step is the synthesis of the CoSb3 micron-sized particles starting from high purity Co powders (99.998%) and Sb shots (99.999). The metallurgical route utilized to prepare a CoSb3 ingot is described in detail elsewhere [19]. The ingot was then crushed into µ-sized particles in an agate mortar. Nanosized ZnO powders were purchased from Alfa Aesar (APS powder, 99% with a measured size of about 100 nm). The micro and nano powders were mixed together during 20 min in a methanol/hexane (3:1) mixture at 0.025M in an ultrasonic bath respecting different weight ratio (x = 2% -4% -8% and 12%) of ZnO. The as-dispersed particles in the liquid were immediately immerged in liquid nitrogen and dried by freeze-drying during 12 hours (FTS-EZ585 system, 17 mTorr, -80°C). Finally, the composite powders were densified by SPS (Dr Sinter-515S) in graphite dies (10.4 mm in diameter) at 600°C during 6 min under 50 MPa. The densities of the obtained pellets were estimated from the mass and the geometry of the samples. The theoretical densities of the composites dcomposite, reported in table I, were calculated using the formula [START_REF] Lenoir | Semiconductors and Semimetals[END_REF] where and dZnO are the densities of CoSb3 (7.64 g.cm -3 ) and ZnO (5.67 g.cm -3 ), respectively [20]. The relative densities (table I) decrease as the ZnO content in the CoSb3 €

d composite = 1 x /d ZnO + (100 -x) /d CoSb 3 € d CoSb 3
matrix becomes higher.

Chemical and structural characterizations and transport property measurements

The crystalline structures were investigated by X-ray diffraction (XRD) performed on a monochromatized CuKα1 Bruker D8 Advance diffractometer equipped with a LynxEye detector. Information on the lattice unit cell was obtained via Rietveld refinement using the Fullprof software [21]. The micromorphology of the samples have been observed by scanning electron microscopy (FEG-SEM, Philips XL-30) in the secondary electron (SE) or backscattered electron (BSE) mode on polished and fracture figures.

The SPS densified cylindrical samples were cut with a diamond wire saw into subparts for independent measurements of their physical properties. The measurement of the thermoelectric properties at low temperatures (2 -300 K) were carried out using the thermal transport option (TTO) of the physical properties measurement system (PPMS, Quantum Design) on bar-shaped samples (typical dimensions 2.5×2.5×10 mm 3 ) which were cut perpendicularly to the pressing direction. The TTO allows sequential measurements of the electrical resistivity, Seebeck voltage and thermal conductivity at each temperature. For this measurement, four copper leads were attached onto the sample with conducting silver epoxy. Special care was taken to ensure good electrical and thermal contacts. A temperature gradient was established by applying heat at one end of the specimen while holding the other end at a constant temperature by maintaining good thermal contact with a low-temperature reservoir.

Calibrated cernox thermometers were attached to the sample between 4 and 5 mm apart. The temperature difference was monitored and the voltage measured at the same positions along the sample. The data collected were corrected for thermal radiation by a modified model implemented in the TTO option software (from version 1.1.4) by Quantum Design. This model differs from that used in our prior work primarily in that it uses a more accurate estimate of the bath temperature around the sample. The modified model results in a higher correction near 300 K thereby explaining the difference observed between our previous and present data sets [22]. Hall coefficient measurements were conducted between 5 -300 K, on the same samples used for TTO measurements, using a five-probe method with the AC transport measurement system option of the PPMS. The electrical contacts were attached onto the samples with silver paste. The magnetic field was varied between ± 5 T to eliminate voltage probe misalignment effects.

For measurements over the 300 -800 K temperature range, the thermopower and the electrical resistivity were simultaneously measured in a ZEM3-ULVAC device under partial helium atmosphere on bar-shaped samples. Thermal conductivity values were obtained from measurements of thermal diffusivity (NETZSCH, LFA 427) and specific heat (NETZSCH, DSC 403 F3) both realized under argon atmosphere. For the thermal diffusivity measurement, separate disks were sectioned from the same individual ingots.

Uncertainties in the electrical resistivity, Hall effect, thermopower and thermal conductivity measurements are at most 5%, 5%, 5% and 8 %, respectively. The results of the low and high temperature measurements match well near room temperature.

Results and discussion

Phase and microstructures

Figure 1 shows the XRD patterns of the CoSb3 + x wt% ZnO after SPS. It can be seen that the matrix (x = 0) is a single-phase material with a minute amount of Sb as a secondary phase. The diffraction pattern is marked by distinct and narrow diffraction peaks. With the introduction of ZnO, new diffraction peaks appear whose intensities become higher with increasing the ZnO amount; those peaks correspond to the ZnO phase. Even for the lowest ZnO content (2 wt%) added, peaks from the ZnO phase can be observed suggesting that ZnO is not finely dispersed in the CoSb3 matrix. The lattice parameter (see table I) has been inferred from Rietveld refinement and was found constant regardless of the ZnO content.

The SEM analysis indicates that the CoSb3 matrix is constituted of particles whose size is about 2-5 µm, as it can seen for example for the 2%ZnO-CoSb3 sample (Fig. 2). This size is neither influenced by the presence of ZnO nor by its content. In addition, large ZnO aggregates are clearly observed at the grain boundaries of the composites, even for low percentages, but isolated aggregates are also present (Fig. 3). The areas where ZnO aggregates can extend up to hundreds of microns as the nominal percentage of ZnO is increased. These irregular micron-clusters have already been observed [14,[23][24][25] in different composites with CoSb3 as the matrix. The average size of the ZnO nanoparticles does not evolve with the SPS treatment and remains of the order of 100 nm. They present a pulverulent aspect (Fig. 2). This can be explained by the low sintering temperature used for skutterudites, too low to sinter ZnO whose sintering temperature is around 1200°C. This aspect likely induces higher porosity that explains the decreasing densities of the samples with the ZnO content (Fig. 3, table I).

Low-temperature thermoelectric properties

The temperature dependence of the thermopower for all the samples studied is shown in wt% ZnO. Further increasing x to 12 wt% then lowers S that only reaches 165 µV/K.

The Hall coefficient was found to be positive for all samples in agreement with the S(T) data. The hole concentration p was calculated from the values of using the singlecarrier formula (e is the elementary charge), whereby the Hall factor was assumed to be equal to 1. p is displayed as a function of temperature in Figure 5 and the room temperature values are listed in table II.

As a first approximation, p can be considered as constant in the temperature range studied whatever the sample is. Surprisingly, these measurements revealed a decrease of p when the ZnO content increases. It is worth mentioning that ZnO is extremely stable and decomposes into zinc and oxygen only at 2348 K [START_REF] Porter | Zinc Handbook: Properties, Processing, and Use in Design[END_REF]. A decomposition of the ZnO nanoparticles during the SPS treatment therefore appears highly unlikely. Further, there is no report in the literature indicating that either Zn or O can substitute for Co or Sb or fill the structural voids. As a consequence, the decrease in the hole concentration with x suggests that holes could be trapped at the grain boundaries where ZnO and/or pores are present. It is known that most ZnO has n-type character due to unintentional incorporation of impurities acting as shallow donors [START_REF] Janotti | [END_REF]. It is thus probable that holes coming from the surface of the CoSb3 grains diffuse into the ZnO barriers to form electron-hole pairs thereby reducing the itinerant carrier density.

The electrical resistivity as a function of temperature is shown in Figure 6 for the CoSb3/ZnO composites. r shows a semiconducting behavior over the 2 -300 K temperature range for all samples. The r values of pure CoSb3 match those reported previously at comparable hole concentrations [28]. It appears that r increases with the addition of ZnO. At 300 K, the values are 1.5 to 12 times higher than that of the reference sample. At low temperatures however, the difference is more pronounced and reaches 3 orders of magnitude

R H R H p = r H / R H e = 1 / R H e r H
at 2 K for the x = 12 wt% sample. As shown below, these results are consistent with mobility data indicating that the hole mobility is strongly degraded at low temperature when the concentration of ZnO increases.

Figure 7 shows the temperature dependence of the Hall mobility for all specimens. For CoSb3 and the composites, increases with increasing temperature up to 300 K. In polycrystalline p-type CoSb3 with comparable grain size, a variation standing for ionized impurity scattering, is usually the dominant scattering mechanism below 300 K [29]. Yet, this law cannot describe satisfactorily our experimental data. A better agreement can be obtained by fitting the data with the equation

where L is the grain size, e is the elementary charge, m * is the effective mass, is the Boltzmann constant and is the height of the energy barrier in the depletion region. This temperature dependence suggests that the grain boundaries dominate the hole diffusion as expected in samples with fine microstructure [30]. As can be seen in Figure 8, which depicts vs , a nearly linear region appears between 150 and 300 K. Fitting this temperature range using Eq.( 3) yields an activation energy = 15 meV. When the average energy of the holes becomes higher than this energy barrier, the conduction is then dominated by thermionic emission (ballistic transport) following a law. This mechanism dominates in this temperature range while at lower temperatures transport is dominated by thermally-assisted tunneling effects.

The data of the composites show a steeper rise in temperature than that of the reference sample. However, the presence of ZnO is detrimental to the mobility of holes,

µ H = R H / ρ µ H T 3/2 µ H (T ) µ H = eL 2π k B Tm * exp - E B k B T " # $ % & ' k B E B ln(µ H T 1/2 ) 1 / k B T E B T -1/2 exp -E B / k B T ( ) µ H (T )
which drastically decreases. The impact of ZnO is more perceptible at low temperatures and all the more pronounced that the ZnO content is high. As for the scattering mechanisms, grain-boundary scattering seems to still play a role in the composites (cf. Fig. 7). This observation seems reasonable since the depletion layers formed at the ZnO/CoSb3 interfaces are expected to behave as energy barriers impeding the conduction of holes between grains.

The values of deduced from the fit (see table II) are in line with those found in prior studies [31] and increases with the ZnO content suggesting that tunneling conduction decreases.

At equilibrium, the barrier height and its width are linked to the density of trapping states and the carrier concentration p by [32] (4) [START_REF] Morelli | [END_REF] where and are the permittivity of vacuum and of CoSb3, respectively. The latter amounts to = 42.4 [33]. Table 2 summarizes the values of and for all the specimens. The density of trapping states seems independent of the ZnO concentration while the width increases with increasing x. Assuming m * = 0.07m0 (m0 is the bare electron mass) [28,34,35],

we estimated the effective grain size using Eq. ( 3) (Table II). The obtained values are consistent with the SEM images (cf. Fig. 2).

The low-temperature thermal conductivity data are plotted in Figure 9. At room temperature, a reduction close to a factor two can be observed for the 12 wt% ZnO composite.

As mentioned above, the electrical resistivity of the ZnO/CoSb3 composites is greater than 4 mW.cm for all x contents. These large values lead to an electronic thermal conductivity, ke, estimated using the Wiedemann-Franz law ( is the Lorenz number and equals

E B E B b N t E B = e 2 N t 2 8ε 0 ε p b = 1 2 N t p ε 0 ε ε N t b κ e = LT / ρ L to 2.44x10 -8 V 2 .K -2
for degenerate electron gas) which never exceeds 2% of the total thermal conductivity k. Hence, the data almost entirely reflect the lattice thermal conductivity kL. The addition of ZnO into CoSb3 does not influence the temperature dependence that remains typical of a dielectric crystal but decreases the values of kL. For x = 2, 4 and 8 wt%

ZnO, this reduction is low but becomes more significant for x = 12 wt%. The decrease of the lattice thermal conductivity in the composite samples may be not only attributed to the large oxide defects localized at the grain boundaries but also to the porosity. However, a quantitative description of the role played by each of these mechanisms was not carried out in the present study but might be an interesting undertaking in future studies.

Figure 10 displays the calculated dimensionless figure of merit ZT versus T. For all samples, ZT increases with temperature. A significant enhancement of the ZT values by 30% at 300 K can be observed for the x = 2 wt% sample. Higher ZnO concentrations efficiently reduce the thermal conductivity but at the expense of a significant increase in the electrical resistivity. Thus, this combination leads to lower ZT values than those of the binary CoSb3 compound in the whole temperature range.

High-temperature thermoelectric properties

The temperature dependences of the thermoelectric power of the binary CoSb3 skutterudite and the ZnO/CoSb3 composites are depicted in Figure 11. The four composites exhibit similar behaviours and values above 500 K. While this fact holds true for the x = 2, 4 and 8 wt% samples below this temperature, a sizable difference appears with the x = 12 wt% sample for which lower values are achieved. When the temperature rises, the S values peak before starting to decrease. The temperature at which the maximum occurs is near 520 K in CoSb3 (Smax = 200 µV.K -1 ) and is shifted to a lower temperature (450 K) when ZnO is added κ (T )

(Smax = 252 µV.K -1
). No influence of the ZnO content on the temperature of this maximum is observed in the present data. Above 600 K, it can be noted that all materials of this series possess similar Seebeck coefficient values. The downturn observed is likely the signature of minority carrier effects. All samples enter the intrinsic domain of conduction.

The temperature dependence of the electrical resistivity is reported in Figure 12. As already highlighted by the low-temperature data, the composites exhibit higher values compared to CoSb3 in the whole temperature range investigated. The values are particularly high in the x = 12 wt% sample (up to 70 mW.cm -1 at 300 K). The data show a semiconducting behavior above 450 K. The presence of minority carriers is likely at the origin of this dependence since both types of carriers inevitably contribute to the electrical conduction. In CoSb3, the curves drop in a steeper way near 500 K compared to the composites. Beyond 700 K, it can be noticed that the data do not merge into a single curve owing to different relative densities as shown by the following relation [START_REF] Landauer | Electrical Transport and Optical Properties of Inhomogeneous Media[END_REF] (7)

where is the electrical conductivity.

The temperature dependence of the total thermal conductivity of the ZnO/CoSb3 composites is shown in Figure 13a. The combination of interfaces, wide defects and porosity contributes to decrease the values of κ in the composites compared to CoSb3. The overall temperature dependence of the thermal conductivity is not influenced by the addition of ZnO nanoparticles. first decreases with temperature, shows a minimum near 650 K and then increases very slightly to reach a value of about 5.0 and 2.4 W.m -1 .K -1 at 800 K, for CoSb3 and 12 wt% ZnO/CoSb3, respectively. This behaviour is in agreement with the results of the literature for CoSb3 [START_REF] Sharp | [END_REF].

ρ(T ) ρ(T ) d σ 100% = 2σ 3d -1 σ κ (T )
Assuming that the Umklapp mechanism is the main phonon scattering mechanism in this temperature range, the lattice thermal conductivity is then expected to vary following a law. This contribution dominates the thermal transport between 300 and 600-650 K, as can be seen in Figure 13b. The enhancement of k beyond 650 K may be due to an additional contribution to the electronic thermal conduction that can be attributed to the presence of the minority carriers resulting in electron-hole pairs (ambipolar contribution). Actually, in presence of electrons and holes, the electronic thermal conductivity can be expressed as [START_REF] Rowe | Thermoelectrics Handbook: Macro to Nano[END_REF] (7)

where and are the contributions due to electrons and holes, respectively, and is the component due to the electron-hole pairs (ambipolar contribution). The Wiedemann-Franz law can be applied for the determination of and by considering the partial electrical conductivities. The ambipolar component, however, depends on the partial electrical conductivities of the electrons and holes as well as on the partial thermoelectric powers. The determination of these parameters requires applying a two-band model. Here, we therefore did not separate the bipolar from the lattice contribution. In the composites, for which the electrical resistivity is higher than for CoSb3, the total thermal conductivity is still dominated by the lattice contribution.

The temperature dependence of the dimensionless thermoelectric figure of merit, ZT, calculated on the basis of the thermal and electrical measurements, is reported in Figure 14 for

CoSb3 and the ZnO/CoSb3 composites. Except for the x = 12 wt% sample for which the high electrical resistivity is detrimental, the presence of ZnO slightly improves the thermoelectric performance of CoSb3, with a maximum of 0.14 reached at 595 K for the x = 2 wt% sample due to the reduction of the thermal conductivity.

Conclusion

1 / T κ = κ el +κ h +κ el-h κ el κ h κ el-h κ el κ h
ZnO nanoparticles were introduced via a new route into the CoSb3 skutterudite matrix to form composite materials and the effect of the presence of these oxide nanoparticles on the thermoelectric properties of the composites were investigated. ZnO agglomerates into irregular micron sized clusters located at the grain boundaries of the matrix. Both holes and phonons are affected by the large oxide defects and the porosity that increase with increasing the ZnO content. Low weight percentage of ZnO (2 wt %) enhances the dimensionless figure of merit with an improvement of up to 36% at 300 K decreasing unfortunately to 6% (within the uncertainty range of measurements) at 595 K, temperature of use of skutterudites. Further optimization might be achieved through improving the synthesis method to obtain a better dispersion of the ZnO nanoparticles and/or by introducing nanoparticles with a smaller size.

Another worthwhile study would be to determine whether the enhancement of the ZT values survives the introduction of a filler atom in the cages of the crystalline structure. Such an attempt was successfully performed with Ag nanoparticles dispersed Ba-filled skutterrudites [38]. The Hall concentrations and mobilities were measured at 300 K. The four other parameters were inferred using Eq. ( 3), ( 4) and (5).

x p (10 18 cm -3 ) (cm 2 .V -1 .s -1 ) (meV) (10

Figure 4 .

 4 S is positive in the binary compound and in the composites suggesting that the holes are the dominant carriers. The positive sign of the Hall coefficient further confirms this hypothesis (not shown here). Regardless of the composition, S rises monotonically with the temperature. Across the entire temperature range, the presence of ZnO inclusions induces R H an increase of the S values. At 300 K, S ranges between 190 and 200 µV/K for x = 2, 4 and 8

Figure 1 :

 1 Figure 1: X-ray diffraction patterns of the SPS densified CoSb3 + x wt% ZnO samples.

Figure 2 :

 2 Figure 2: Fracture image of the CoSb3 + 2 wt% ZnO sample.

Figure 3 :

 3 Figure 3: SEM (back-scattered electrons) images of the CoSb3 + x wt% ZnO samples. The

Figure 4 :

 4 Figure 4: Temperature dependence of the thermopower S of the CoSb3 + x wt% ZnO

Figure 5 :

 5 Figure 5: Temperature dependence of the hole concentration p.

Figure 6 :

 6 Figure 6: Electrical resisitivity r as a function of temperature of the CoSb3 + x wt% ZnO

Figure 7 :

 7 Figure 7: Hall mobility as a function of temperature.

Figure 8 :

 8 Figure 8: (a) vs for the composite materials. (b) Best fit to the data

Figure 9 :Figure 10 :

 910 Figure 9: Temperature dependence of the thermal conductivity k of the CoSb3 + x wt% ZnO

Figure 11 :

 11 Figure 11: Temperature dependence above room temperature of S for CoSb3 and the four

Figure 12 :

 12 Figure 12: Temperature dependence at high temperatures of r of the CoSb3 + x wt% ZnO

Figure 13 :

 13 Figure 13: k as a function of temperature for CoSb3 and the four ZnO/CoSb3 composites: a)

Figure 14 :

 14 Figure 14: Temperature dependence of ZT of CoSb3 + x wt% ZnO.

Figure 3 25 Figure 4 26 Figure 5 Figure 6 28 Figure 7 Figure 9

 25426562879 Figure 3

Figure 10 32 Figure 11 33Figure 12

 321112 Figure 10

Table captions Table I :

 captionsI Theoretical density dcomposite, relative density d and lattice parameters a of CoSb3 + x

	wt% ZnO.
		dcomposite	d	a
	x	(g.cm -3 )	(%)	(nm)
	0	7.64	98	9.0350
	2	7.59	91	9.0353
	4	7.54	91	9.0342
	8	7.43	89	9.0341
	12	7.33	83	9.0357

Table II :

 II Hole concentration p, Hall mobility , energy barrier , density of trapping states , width of the energy barrier and calculated grain size of CoSb3 + x wt% ZnO.

ACKNOWLEDGMENTS

Authors acknowledge the support by the European Network of Excellence CMA (Complex Metallic Alloys).